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Executive Summary 


In October 2010, Jefferson County Public Schools (JCPS) was awarded a four-year, 1 five- 
million dollar development grant by the U.S. Department of Education through the inaugural 
Investing in Innovation (i3) program competition. JCPS’ project, Making Time for What Matters Most, 
aims to improve student achievement, narrow achievement gaps, strengthen students’ college 
readiness skills, and increase the percentages of students who graduate and go on to college. In 
order to achieve these overarching aims, JCPS defined three goals: (1) provide structures and 
supports to facilitate student mastery of academic material and successful completion for all core 
courses in one year or less; (2) provide a range of personalized supports to students to increase 
engagement in school and promote college readiness; and (3) improve teachers’ pedagogical and 
student support practices to maximize the effectiveness of increased learning time. 

JCPS’ district high school leadership selected six persistently low-performing high schools 
for the project: Academy @ Shawnee, Fern Creek Traditional High School, Moore Traditional High 
School, Valley Traditional High School, Waggener High School, and Western High School. These 
schools were chosen because they serve higher proportions of minority and economically 
disadvantaged students relative to both the county and district overall. For the 2014-2015 school 
year, a mean of 56% of the students attending the i3 schools were racial/ ethnic minority students, 
compared to 48% of high school students across the district. All of the i3 schools are Title I schools 
with the exception of Fern Creek Traditional High School. In addition, across the i3 high schools, 
71% of students were enrolled in the free and reduced-price meal (FRPM) program, ranging from 
65% to 79%, compared to 55% of high school students across the district. 

McREL International serves as the external evaluator for the JCPS i3 project and is 
conducting comprehensive, mixed-methods process and outcome evaluations. The evaluations 
being performed by McREL incorporate data from surveys, interviews, extant documents, and 
student education data. The process evaluation provides feedback to JCPS and the targeted schools 
to enable quality improvement, identify successes and challenges, and document the structures and 
processes related to project implementation. The outcome evaluation examines the impact of the 
intervention components or strategies on proximal, intermediate, and long-term student and school 
outcomes. 

This report focuses on the Year 5 evaluation of the project, which occurred between 
October 1, 2014 and September 30, 2015. When appropriate, data reported in the evaluation reports 
for Years 1 through 4 (Nixon et al., 2012; Donahue, Fryman, Ho, & Stone, 2012; Donahue, Ho, 
Fryman, & Stone, 2013; and Donahue, Ho, Stone, & Lane, 2014) are presented to show the 
i3 program’s progress toward the end-of-project targets, compared to progress made in those grant 
years. This Year 5 report (1) provides an overview of the project’s goals and objectives; 

(2) describes the evaluation design and methods; (3) reviews the progress of Year 5 implementation 


1 In fall 2014, the U.S. Department of Education approved a no-cost extension to the project to extend it for a fifth year. 
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based on specific indicators; and (4) provides conclusions and recommendations for each project 
goal. 

Year 5 Project Conclusions and Recommendations 

Conclusions pertaining to the six i3 schools’ overall performance on the objectives in Year 5 
are organized around the project’s three goals and are derived from analyses of data collected during 
the fifth year of the project. Recommendations for each goal are based on a review of the data and 
the conclusions. The recommendations are offered for JCPS staff to consider as they make 
determinations regarding the sustainment of project components once the grant ends. 

Project Goal I : Student Mastery of Academic Material 

The first goal of the project is to provide stmctures and supports to facilitate students’ 
mastery of academic material and successful completion of core courses in one year or less. 

Progress on this goal is measured through interviews with the counselors to learn how they assign 
students to appropriate classes. The percentage of students who pass the core courses in which they 
are enrolled each year is also analyzed by the evaluation team to determine progress on this goal. 
Additionally, teachers and students were asked questions regarding their perceptions of the 
scheduling process on surveys administered by the evaluation team. 

Course Assignment Protocol and Monitoring of Progress. Counselors have 
developed processes in their schools for making student placement decisions, including visiting 
middle schools to work with incoming freshmen. To place upperclassmen, the counselors use a 
variety of resources, including classroom grades, assessment data, and students’ abilities and 
interests. 

The counselors monitor student progress by engaging with teachers and students, and in 
some cases, parents. Students may move into a more or less challenging course upon their request 
or based on a recommendation by a teacher or parent. Counselors will review all student data to 
ensure that the new placement best meets the needs of the student. 

While counselors see the benefits of providing time for struggling students to make up 
credits, they also see disadvantages for both struggling and accelerated students, such as fewer class 
choices because of the five -period schedule rather than a typical seven-period schedule. There are 
concerns that struggling students take advantage of the opportunity to have multiple chances to pass 
a course. Although the trimester was described as “irrelevant” for advanced students by one 
counselor because they are primarily enrolled in year-long courses, another counselor was concerned 
that the trimester does not allow accelerated students the opportunity to take electives or career 
pathway courses. Based on the evaluation findings, McREL evaluators make the following 
recommendations: 

• Request that counselors and other affiliated school personnel document their scheduling 
and monitoring processes. From these submissions, create a “Best Practices” resource 
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that will provide a multitude of ideas that schools can adapt to fit into their own 
contexts. 

• Bring schools together to share their successes and challenges. During this time, have 
district staff present information on who can address questions that master schedulers 
and counselors may have that relate to scheduling (e.g., Infinite Campus) or student 
placement. 

• Develop case studies of specific programs (e.g., Schools of Study) which may warrant 
further study and be of interest to the schools throughout the district. 

• Have the counselors facilitate meetings with the faculty at their schools to learn more 
about why teachers indicated on the teacher survey that the schools may not always have 
the best placement strategies for students. 

• Create time for school staff to discuss how the current scheduling process impacts both 
struggling and accelerated students as they were divided on whether the current 
processes benefitted these students. Listening to different perspectives may encourage 
and create new ideas that can benefit all students, regardless of their academic status. 

Core Course Pass Rates. Overall, approximately 82% ofi3 students passed their core 
courses. Across the subject areas, the pass rates were higher in English (86%) and social studies 
(84%) than in math (79%) and science (77%). Although student pass rates varied across different 
subjects and core courses, one school (i.e., Moore Traditional High School), in particular, had higher 
pass rates than did others. While the majority of schools stmggled to meet the Year 5 targets, 

Moore Traditional High School exceeded the Year 5 target (1% above the target) in social studies 
and was only 0.6% shy of the science target. 

Findings of subgroup analyses revealed that trimester scheduling seemed to have positive 
effects in English, math, social studies, and some sub-areas of science (i.e., life science and 
earth/ space science) in comparison with the traditional two-term courses. Additionally, 
achievement gaps measured by course pass rates continued to be observed among the subgroups. 

In particular, female students had higher passing rates as compared to male students across all 
subject areas and across the majority of the core courses. Nevertheless, consistent with national 
trends, students from certain racial/ethnic (i.e., Black/ African American) and socioeconomically 
disadvantaged (i.e., those with FRPM status) groups fell behind their counterparts. Such differences 
were more pronounced between White students and Black students and between students from 
middle- and upper-income families (i.e., students with paid meal status) and low-income families 
(e.g., students with FRPM status). These gaps have been well documented in the literature as well as 
in the previous project reports. Yet, it is interesting to observe that in Year 5, Hispanic students 
demonstrated higher pass rates in math, science, social studies, and the majority of the core course 
areas than they have in past years. Based on the evaluation findings, McREL evaluators make the 
following recommendations: 

• Identify, develop, and implement strategies to close achievement gaps among the 
subgroups, specifically the ethnicity/ race and FRPM subgroups. 


XII 


• Examine why specific courses (e.g., Geometry, Physical Science, and Earth/Space 
Science) tend to have lower pass rates than other core courses. Determine which 
strategies teachers may need to utilize to help increase pass rates. Provide professional 
development for teachers to learn how to effectively implement these instructional 
strategies in their teaching. 

• Study the variations of the pass/ fail rates across the schools to determine why some 
schools seem to have higher pass rates than others. Use data from this report and/ or 
school data to create individual school profiles to look at trends, ask more in-depth 
questions, and gain a full understanding of what is happening at each school, specifically 
in terms of the achievement gaps. Develop an action plan based on these findings to 
support the school in closing the achievement gaps. 

Academic Challenge, Self-Efficacy, and Support of Learning from Teachers. 

Overall, the i3 schools showed a slight increase (2%) in the mean percentage of students’ 
perceptions of academic challenge from Year 4 to Year 5. When compared to the end-of-project 
target, i3 schools, on average, were 8% below the goal of 75% agreement. The examination of 
variations across the i3 schools revealed that four schools (i.e.. Fern Creek Traditional High School, 
Moore Traditional High School, Valley High School, and Waggener High School) showed minor 
increases from Year 4 to Year 5; yet, the other two schools (i.e., Academy @ Shawnee and Western 
High School) showed minor decreases from Year 4 to Year 5. Further, results of subgroup analyses 
revealed that students’ perceptions of academic challenge differed by their limited English 
proficiency (LEP) status, grade level, and race/ethnicity. 

Teachers tended to agree that the school provided students with academic challenge. A 
majority of teachers agreed that the students seemed to enjoy school and that high grades are viewed 
as an indicator of mastering high standards. However, a third also agreed that students were bored 
at the school and just over half agreed that students do well at their school. 

Responses regarding student engagement were also positive. Teachers agreed that they 
challenge their students academically and help disenfranchised students feel more connected to 
school. They also tended to agree that the teachers in their school encourage students to go beyond 
stated expectations and involve the students in setting those expectations. However, a majority of 
teachers disagreed that they involve students in developing criteria for assessing their assignments 
and that they work to involve students in planning lessons. Thus, it appears that teachers still 
struggle with ways of encouraging student voice in the classroom. Based on the evaluation findings, 
McREL evaluators make the following recommendations: 

• Investigate variations across the schools to identify strategies and practices that are 
working in some schools but not working in other schools. Experiment with modifying 
strategies and practices for the respective schools. 

• Conduct focus groups with small groups of students across the grade levels to learn 
more about what a “challenging” curriculum means to them, and what they believe 
constitutes “meaningful” homework assignments. 
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• Facilitate conversations with the leadership at each of the i3 schools to review the data 
regarding students’ perceptions of academic challenge, which differed across the schools 
and may suggest that some schools need more support than others. Develop an action 
plan with each school to improve students’ perceptions of academic challenge. 

• Encourage building leadership to have conversations with teachers about how to 
develop an engaging and challenging curriculum. If possible, allow students to join in 
those conversations in order to have their voices heard and to establish a new type of 
relationship with teachers and administrators. 

Project Goal 2: Student Engagement in School and College Readiness 

The second project goal is to provide a range of personalized supports to students to 
increase engagement in school and promote college readiness. The primary strategy is to create 
College Access Time (CAT) advisory periods for students that are designed to focus on college 
readiness. In addition to CAT implementation data, 10th grade ACT PLAN® (hereafter referred to 
as PLAN) scores and 11th grade ACT® scores were analyzed to measure students’ college 
preparedness. Postsecondary transition rates were examined through StudentTracker data. 

Students’ perceptions about their own engagement were measured through the Comprehensive 
School Survey (CSS) and 2015 Student Survey data. 

CAT Implementation. To support college readiness for JCPS students, all project 
schools hired a College Access Resource Teacher (CART) to design and plan the CAT advisory 
periods to increase students’ understanding and interest in attending college. Being that the CARTs 
have the autonomy to plan and design CAT advisory periods within each project school, they have 
implemented projects and have curricula planned out for the school year. However, they still 
struggle with having 100% of the teachers buy into the process and engage students. Further, a few 
of the CARTs mentioned that student apathy is an issue. The CARTs also had limited time and 
opportunity to provide training to their own staff, and there were limited opportunities for them to 
receive training for themselves. Based on the evaluation findings, McREL evaluators make the 
following recommendations: 

• Allow time for the CARTs at the i3 schools to visit other i3 schools (or other district 
schools) to monitor, assess, and provide feedback on how to strengthen each other’s 
programs. This would allow for the creation of a community of learners to be built 
within the i3 project schools, which could be expanded to other CAT programs in the 
district. 

• Provide training at the district level for all advisors so those assigned as advisors can 
continue to see the value of the program and increase their buy-in. 

• Encourage building administrators to allow time for the CARTs to provide training to 
advisors at their schools. This may also help administrators who have not fully bought 
into the program to become more aware of its value and to help build support for CAT. 
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• Create more structured guidelines for what a CAT advisory period should look like. 
Although the CARTs appreciate the autonomy afforded to them, stmcture and CAT 
standards would help them reach targeted goals. 

• Share information with the CARTs on funding resources, particularly for schools where 
the CART position may not be full-time in the future. 

Students’ Knowledge and Use of College Readiness Skills and Postsecondary 
Transition. Based on students’ ACT and PLAN outcomes, a finding from this evaluation was that 
the majority of i3 students are not college ready, using ACT and PLAN benchmarks as an indicator 
of college readiness. Compared to Year 4 findings, the performance of the i3 schools declined in all 
subject areas in Year 5. Although the schools still have not reached the project targets, the gaps 
were reduced across all subject areas. Gaps between performance and the end-of project goals range 
from 13 to 19 percentage points. 

The percentage of i3 students meeting or exceeding the ACT benchmarks decreased in 
English and reading from Year 4 to Year 5 but increased in math. Overall, the gap to the end-of- 
project target ranges from 6% to 14%, with the largest gap in reading, followed by science, math, 
and English. 

In regards to postsecondary transition, there was an overall increase in the percentage of 
students enrolling in a postsecondary institution from 2013 (38%) to 2014 (45%). Although the 
target goal of 55% was not reached, there was an increase of 4% from the baseline to 2014. It 
should also be noted that of those students who did enroll in postsecondary institutions, 

86% attended public institutions, 53% attended four-year institutions, and 90% attended an 
institution in Kentucky. Based on the evaluation findings, McREL evaluators make the following 
recommendations: 

• Review instructional practices to determine why student scores are still not reaching the 
PLAN and ACT benchmarks, specifically in the areas of math and science. Provide 
professional development to teachers who may benefit from new pedagogical 
instruction. 

• Establish a time for i3 school administrators, teachers, and CARTs to share their lessons 
learned or best practices of what has or has not worked in their schools to improve 
academic achievement. Some schools have made more progress than others. Start with 
a team at those schools to learn more about what they have done to achieve their current 
accomplishments. 

• Strategize to further reduce achievement gaps among subgroups, particularly 
racial/ethnic groups and students qualifying for FRPMs. 

• Continue to examine postsecondary transition data and address questions regarding how 
rates can continue to increase. 

• Encourage the CARTs to conduct exit interviews with seniors to find out why they made 
particular choices. CAT should be a time for students to set goals and work with their 
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advisor to ensure that they achieve those goals. Track the progress of students; if 
students have developed relationships with their advisors over the course of the school 
year, encourage both the advisor and student to send an e-mail in late summer or early 
fall to track what the student is doing and why that student chose the path that he or she 

did. 

Student Perceptions of School Climate. Multiple CSS items were used to examine 
students’ perceptions of (1) support for learning (1 item), (2) adult social support (5 items), (3) peer 
social support (1 item), and (4) school connectedness (3 items). Overall, students reported more 
positive than negative perceptions across all constructs. Two items were also selected from the CSS 
to assess student engagement. About one third of students indicated that they were engaged in 
sports or clubs at school. 

Student perceptions also varied by school. Overall, Fern Creek Traditional High School 
reported higher levels of support for learning, adult social support, peer social support, and school 
connectedness than did all other i3 schools. Student perceptions also differed by students’ 
demographic characteristics. For example, male students reported higher levels of peer social 
support and school connectedness than did female students; yet, female students reported a higher 
level of adult social support than did male students. Students from higher grade levels reported 
higher levels of support for learning, adult social support, and school connectedness. Students with 
LEP also reported higher levels of support for learning, adult social support, peer social support, 
and school connectedness than did students without LEP. Based on the evaluation findings, 
McREL evaluators make the following recommendations: 

• Review the school culture data with the i3 schools. Develop action plans on how the 
staff at each school can continue to improve their climate. 

• Continue to develop a culture within the schools that provides peer-to-peer support to 
ensure there is a respectful culture between students as well as adults. 

• Work with building administrators to develop stronger relationships with families and 
community members. Continue to emphasize afterschool programs and activities while 
ensuring that all students have access. 

• Investigate why grade levels differed across the various CSS constructs. For example, 
why do 1 2th graders believe there is more support for learning than ninth graders? If 
there are certain activities happening at a particular grade level to make them feel more 
supported, how can those activities be replicated in other grade levels? 

• Explore why students with LEP tend to have more positive perceptions of school 
culture. If it is because of special services they receive, brainstorm ways in which other 
students can receive more personalized services to meet their individual needs. 

Fidelity. No schools on this component reached a threshold of implementation. However, 
it should be noted that the indicators are based upon JCPS’ original proposal on what they thought 
was achievable. Once the i3 schools began to implement the CAT advisory periods, they found that 
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adjustments had to be made to fit CAT within the school schedule. For example, not all of the 
schools could implement a 55-minute CAT period due to scheduling conflicts. Other schools 
decided that they only wanted to offer CAT biweekly in order to increase academic time. Thus, the 
goals originally set for CAT in the proposal were not feasible for all participating i3 project schools. 
Based on the evaluation findings, McREL evaluators make the following recommendations: 

• Discuss the fidelity components with school leaders and teachers so they have a clear 
understanding of the expectations of CAT (e.g., how many times per week the advisory 
periods should be held and how many minutes they should last). 

• Provide relevant, useful, and timely professional development sessions for CARTs, 
especially on ways in which they can provide training to advisors in their schools. Focus 
on how they can train advisors to engage and motivate students. 

• Develop a common walkthrough tool and provide guidance to the CARTs on how to 
provide feedback to the advisors after they have been observed by the CART. Establish 
a continuous improvement model for CARTs to use with their advisors in order to 
strengthen the relationships between advisors and students. 

Project Goal 3: Teacher Pedagogical and Student Support Practices 

The third project goal is to improve teachers’ pedagogical and student support practices to 
maximize the effectiveness of increased learning time. The strategy to reach this goal is to ensure 
that content-based and cross-disciplinary professional learning communities (PLCs) meet regularly 
during the school year. The increase in PLCs should lead to improvements in teachers’ perceptions 
of collaboration and self-efficacy. In terms of instructional practice, teachers’ perceptions of how 
their own instmctional practices increase student academic engagement and challenge should 
improve, as well as their perceived ability to identify and appropriately respond to students’ unique 
academic and social needs. 

PLC Meetings. Findings showed that more time is allocated to content-based PLCs than 
cross-disciplinary PLCs. Based on survey responses, on average, the content -based PLCs met 
weekly across the project schools while the cross-disciplinary PLCs met less frequently. Survey 
results showed that half of the i3 project schools hosted cross-disciplinary PLCs monthly while the 
other three schools met once a trimester. Principals reported that they were pleased with the 
implementation of both types of PLCs at their schools. Typical challenges that were shared included 
lack of time and monetary resources to continue to support the PLCs. Based on the evaluation 
findings, McREL evaluators make the following recommendations: 

• Provide continued assistance to schools to help them find funding for the provision of 
stipends to teachers for participating in activities outside of their normal duties or to pay 
for substitutes so teachers can participate in “data days”. 

• Allow principals and teacher leaders to visit other high schools to observe their PLCs 
and other effective collaboration practices. 
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• Communicate with principals and teachers to let them know how the PLCs will be 
continued without the support of the i3 grant as concerns were raised about this issue. 

• Review the target benchmarks to determine if modifications need to be made to better 
reflect how the PLCs operate in practice. 

• Establish districtwide guidelines on how often content-based and cross-disciplinary 
PLCs should meet. Ensure that each school understands how JCPS defines a PLC so 
school personnel consistently record meetings. 

• Systemically collect information about the how schools facilitate PLCs. Create a “Best 
Practices” document that describes effective practices across schools. Facilitate a 
meeting with the schools to share “Best Practices” and help them learn from one 
another. 

Increase Teachers’ Perceptions of Collaboration. Teachers tended to have higher 
levels of agreement on survey items regarding teachers supporting one another, collaborating, and 
working together. This may be a result of the PLCs, where time is devoted for teachers to work 
collaboratively to discuss students’ progress and challenges in a supportive environment. Teachers 
were less likely to agree on items regarding resources and time. A large percentage of teachers 
disagreed with the statements about having enough time to collaborate with colleagues to improve 
instruction and the ability to access expertise and sufficient resources in a timely fashion. Another 
survey item that did not have a high level of agreement was “Teachers in this school trust each 
other.” However, it takes time to develop tmst and a tmly collaborative culture, and it seems that 
the schools are continuing their work in building a trusting culture within their schools. Based on 
the evaluation findings, McREL evaluators make the following recommendations: 

• Continue to provide professional collaboration opportunities for teachers with the 
inclusion of trust exercises. Although collaboration is building within the schools, results 
for this area indicate that trust issues remain within the schools. 

• Work with building administrators to ensure that the schools have adequate time and 
resources for collaborative planning and PLCs. The district can provide ideas and 
strategies to help the schools meet the scheduling needs of both students and teachers. 

• Survey teachers and building administrators to learn more about what resources may be 
lacking or what resources they believe will help them improve their teaching practices. 

• Communicate with the schools about how collaboration will continue to be supported 
by the district once the i3 funding is gone. 

Improve Teachers’ Perceptions of Their Own Self-Efficacy with Respect to 
Content-Based Knowledge for Teaching. In general, teachers tended to have high levels of 
self-efficacy as over 80% of them responded positively to survey items related to this constmct. The 
items that did not score as highly were about pressure to achieve immediate results. Only a quarter 
of the teachers responded that they never or seldom worry about being criticized if positive results are 
not readily available and over half (59%) are sometimes or almost always discouraged from trying new 
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approaches because of the emphasis on success. Based on the evaluation findings, McREL 
evaluators make the following recommendations: 

• Provide training sessions for school staff to help them look more positively at data so 
they can measure progress by the “smaller” successes and look at trends over time rather 
than immediate results. 

• Document innovative teaching strategies being used with success. Build time into 
schedules to allow teachers to share innovative practices that have led to student success. 

• Work with administrators on ways to motivate staff and develop a culture that does not 
include fear of being criticized and one that promotes innovative techniques, as long as 
results are being measured and continuously improved. 

Improve Teachers’ Perceptions of Their Instructional Practices in Order to 
Increase Student Academic Engagement and Challenge. Teachers tended to rate survey 
items that focused on challenging students and working with disenfranchised students higher than 
items related to students being actively involved in planning lessons, setting expectations, and 
developing criteria for lessons. It appears that teachers have fewer tendencies to actively involve 
students in planning lessons, although teachers perceived themselves to be challenging students 
academically and setting high expectations for them. Based on the evaluation findings, McREL 
evaluators make the following recommendations: 

• Continue to work with staff to encourage students to go beyond the expectations set by 
teachers. Provide training to help motivate students who may have apathetic tendencies. 

• Encourage classroom teachers to collaborate with the students’ advisors to encourage 
and motivate them to go beyond classroom expectations. 

• Assist teachers in developing lesson plans that will encourage an increase in students’ 
involvement (i.e., student voice) within the classrooms. 

• Develop protocols that teachers can use in the classroom as they work to increase 
student voice. Because this may be a different teaching model than what teachers are 
typically used to, the district may need to expend time and resources to gain buy-in from 
teachers in understanding how this type of teaching strategy may improve student 
outcomes. 

• Conduct focus groups with a small sample of students to collect their input on how best 
to increase their voice and active engagement in terms of lesson implementation. Have a 
pilot group of teachers implement those strategies to document what works and share 
their successes and challenges with other teachers in their building and throughout the 
district. 

Improve Teachers’ Perceptions of Their Ability to Identify and Appropriately 
Respond to Students’ Unique Academic and Social Needs. Overall, school personnel 
reported feeling more positive than negative about the adult social support and support for learning 
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items. However, one item, in particular, received a much lower rating as compared to all other 
items, “Teachers at my school assign meaningful homework on a regular basis” (55% positive 
responses). Additionally, school personnel’s ratings of adult social support and support for learning 
differed by school. School personnel ratings also differed by the demographic characteristics of the 
staff; however, the patterns are inconsistent across the two different subscales. Specifically, in terms 
of support for learning, male and White staff members reported higher ratings than did female staff 
and staff from racial/ ethnic minority backgrounds. Regarding adult social support, staff with more 
than two years of teaching experience perceived a higher rating than did staff with less than one year 
of teaching experience. Based on the evaluation findings, McREL evaluators make the following 
recommendations: 

• Observe teachers using the district observation protocol to determine if they are 
effectively differentiating instruction for all students. 

• Provide professional development on differentiated instmction and how to personalize 
instruction for individual students to teachers who may be struggling in this area. 

• Conduct focus groups or interviews with students to learn more about how they think 
teachers could provide individualized support for them. 

• Encourage dialogue between classroom teachers and students’ assigned advisors to 
further explore how instmction can be individualized for each student. 

PLC Implementation Fidelity. No schools on this component reached a threshold of 
implementation. However, the indicators are based upon the original proposal. Project staff may 
want to reconsider whether or not these are appropriate targets given how the schools ended up 
structuring their PLC sessions. Based on the evaluation findings, McREL evaluators make the 
following recommendations: 

• Create a feedback mechanism to monitor the extent to which teachers implement 
instructional strategies discussed within the PLCs. This may be accomplished through 
the use of an administrator observation form or a district feedback form to gather more 
data about how the work of the PLCs are being implemented in the classroom. 

• Continue to support cross-disciplinary PLCs by providing staff with time to participate 
and resources. Document how the cross-disciplinary PLCs are functioning by 
conducting focus groups or interviews with leaders of those PLCs. Share the findings 
with the i3 schools that may be struggling to implement this component. 
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Project Overview 


Goals and Objectives 

In October 2010, Jefferson County Public Schools (JCPS) was awarded a four-year 2 3 
development grant, totaling five -million dollars across the four years, by the U.S. Department of 
Education (ED) through the inaugural Investing in Innovation (i3) program competition. 1 JCPS’ 
project, Making Time for What Matters Most, aims to improve student achievement, narrow 
achievement gaps, strengthen students’ college readiness skills, and increase the percentage of 
students who graduate and the percentage who go on to college. In order to achieve these 
overarching aims, JCPS defined the following project goals and objectives: 


Goal 1: Provide structures and supports to facilitate student mastery of academic material 
and successful completion for all core courses in one year or less. 


Objective 1A : 


Objective IB : 

Objective 1C : 
Objective ID : 
Objective IE : 

Objective IF : 


Develop assignment/ reassignment protocols such that students are provided 
options of acceleration, remediation, credit recovery, and increased time for 
study based on individual students’ progress and needs. 

Improve ongoing monitoring of student course progress to enable rapid 
response to individual students’ academic needs. 

Increase students’ perception of academic challenge. 

Increase students’ academic self-efficacy. 

Increase the number of students who successfully pass core courses in one 
year. 

Increase students’ perceptions of support for learning from teachers. 


Goal 2: Provide a range of personalized supports to students to increase engagement in 
school and promote college readiness. 


Objective 2A : 


Objective 2B : 
Objective 2C : 
Objective 2D : 


Ensure focused, evidence-based use of College Access Time (CAT) to 
advance college-ready skills, monitor student progress, and increase students’ 
sense of affiliation with adults and peers. 

Increase students’ knowledge of and use of college-ready skills and habits. 
Improve students’ sense of affiliation with adults and peers. 

Strengthen peer-to-peer relations and perceptions of peer support, 
particularly within career themes. 


2 In fall 20 1 4, JCPS was awarded a no-cost extension by ED to continue the project for a fifth year. 

3 The project was submitted for funding as an i3 development project under Absolute Priority 4, “Innovations that Turn 
Around Persistently Low-Performing Schools.” In addition, the district specified one of four competitive priorities, Competitive 
Preference Priority 6: Support College Access and Success. 
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Goal 3: Improve teachers’ pedagogical and student support practices to maximize the 
effectiveness of increased learning time. 


Objective 3 A : 
Objective 3B : 
Objective 3C : 
Objective 3D : 


Increase collaboration among teachers within and across content areas in 
school. 

Increase teachers’ perceptions of self-efficacy with respect to content-based 
knowledge for teaching. 

Improve teachers’ instructional practices with respect to engagement and 
challenge level in core content instructional time. 

Increase teachers’ knowledge and attention to unique student needs and 
support during instruction and CAT. 


A logic model designed to graphically display how the components of the intervention work 
to impact student achievement is provided in Figure 1. Overall, a large percentage of JCPS students 
have been dropping out of high school, and, of those who do graduate, too few have been 
adequately prepared for higher education. The i3 project proposed three interventions that, when 
taken individually, have minimal impact, but when implemented simultaneously have the potential to 
make substantial high school changes. These interventions included expanding learning time by 
developing a master schedule based on a trimester system, increasing the preparation of students for 
postsecondary education by implementing schoolwide advisory periods with trained staff to provide 
guidance, and increasing collaboration among teachers. As depicted in the logic model, the 
intermediate outcomes of integrating the three interventions in the six JCPS high schools that were 
selected to receive i3 funds are to develop, implement, and sustain a district model that provides 
structured time for student acceleration, remediation, and college advising, as well as time and 
support for teachers to plan and learn together. The long-term goal of JCPS is to prepare all 
students in the district to graduate from high school and be adequately prepared for college or 
postsecondary careers. The logic model defines the inputs, outputs, and outcomes for each 
intervention and defines how the interventions will ultimately lead to a decrease in the achievement 
gap, a decrease in the drop-out rate, and an increase in college -ready graduates. 

Target Population 


The project is situated within JCPS, a large urban district with roughly 100,000 students, 
making it among the largest 30 districts in the nation. According to 2010 census data, Jefferson 
County’s total population was 741,096. Approximately 73% of the county’s residents were White, 
21% Black, and 9% Other. Moreover, approximately 21% of all families with children and 25% of 
all children younger than 18 years of age in the county lived below the poverty line. During the 
2014-2015 school year, JCPS’ high school students were 44% White, 44% Black, 8% Hispanic, and 
4% Other. In addition, nearly three quarters (71%) of the students were enrolled in the free and 
reduced-price meal (FRPM) program, which serves as an indicator of those students who are of low 
socioeconomic status. Relative to the county as a whole, the school system serves higher 
proportions of minority and economically disadvantaged students. 
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OUTPUTS 


GOALS 


INPUTS 


Activities 


Participation 


OUTCOMES 

Short/M edium-Term 


Long-Term 


Flexible Scheduling 

• Master schedulers are assigned 
to schools 

• Five 70-minute courses for 
three 1 2-week trimesters 

• Students assigned to courses 
based on data (e.g., academic 
history and assessment scores) 

• Counselors use freshmen 
recommendations provided by 
district 


• Master schedulers 

• Five 70-minute 
courses per day for 
each of three 

1 2-week trimesters 





JCPS provides 
recommendations for 
incoming freshmen 
Master schedulers and 
counselors assign 
students to 
appropriate courses 


• Master schedulers 
complete scheduling 
process 


Increase the number of 
students who 
successfully pass core 
courses in one year 
Increase students’ 
perceptions of 
academic challenge, 
their own academic 
self-efficacy, and 
support for learning 
from teachers 



Achievement 

gap narrows 

College-ready 

graduates 

increase 

Drop-out rates 

decrease 


Professional Learning 
Communities (PLCs) 

• Frequency of content-based 
PLCs in core subjects 

• Duration of content-based 
PLCs in core subjects 

• Frequency of cross-disciplinary 
PLCs 


• Content-based PLCs 

• Cross-disciplinary 
PLCs 


• Content and cross- 
disciplinary PLCs 
implemented in 
schools 


• Teachers and other 
school staff become 
PLC members 


Increase teachers’ 
perceptions of 
collaboration among 
teachers, self-efficacy, 
instructional practices, 
and response to 
student needs 


Figure I. Logic model. 
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JCPS’ district high school leadership selected six of its persistently low-performing high 
schools to participate in the project: Academy @ Shawnee, Fern Creek Traditional High School, 
Moore Traditional High School, Valley Traditional High School, Waggener High School, and 
Western High School. The project schools varied in enrollment from 504 to 1,546, with a mean of 
approximately 959 students. Further, these schools serve higher proportions of minority and 
economically disadvantaged students relative to both the county and district overall. For the 
2014-2015 school year, a mean of 56% of the students attending the i3 schools were racial/ethnic 
minority students. All schools are Title I schools with the exception of Fern Creek Traditional High 
School. Furthermore, across the i3 high schools, 71% of students were enrolled in the FRPM 
program, ranging from 65% to 79%. Table 1 summarizes the characteristics of the six high schools 
participating in the JCPS i3 grant for 2014-2015 school year. 


T able I . Project School Characteristics for 20 1 4-20 1 5 


Academy F ern 

School a @ _ - Moore Valley Waggener Western Mean 


Total # of Enrollment 


1,546 

1,028 

1,100 

766 

810 

959 

% White 

43.8% 

46.4% 

47.2% 

60.1% 

40.1% 

24.8% 

43.7% 

% Black 

44.9% 

37.3% 

33.2% 

32.8% 

46.0% 

68.4% 

43.8% 

% Hispanic 

7.9% 

8.8% 

14.6% 

4.2% 

9.7% 

4.1% 

8.2% 

% Other 

3.4% 

7.5% 

5.0% 

2.9% 

4.3% 

2.7% 

4.4% 

% Enrolled in FRPM 

77.8% b 

64.7% 

64.5% b 

69.2% 

68.7% 

78.5% 

70.6% 

Graduation Rate (2014) c 

72.5% 

82.8% 

87.8% 

77.9% 

88.0% 

8 1 .6% 

8 1 .8% 

Transition to Adult Life 
(20 1 4) d 

98.9% 

92.8% 

98.2% 

58.0% 

8 1 .5% 

9 1 .7% 

86.9% 


a Data for all characteristics are from the JCPS School Profiles, available at 
http://www.jefferson.kl 2.ky.us/Departments/AcctResPlan/Profiles/index.html. 

b Because the Academy @ Shawnee and Moore Traditional High School serve students in grades 6-12, this percentage 
represents all grade levels and is not disaggregated by middle and high school. 
c This is the latest available data for the graduation rate in JCPS. 

d The Successful Transition Rate includes percentages of students who enrolled in college or a vocational/technical school, 
enlisted in the military, became employed, or are employed and going to school. 
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Evaluation Overview 


McREL International is conducting external process and outcome evaluations for JCPS’ 
Making Time for What Matters Most i3 project. McREL evaluation staff have actively engaged key 
JCPS stakeholders in the design, implementation, and interpretation of evaluation findings to 
increase (a) buy-in of school personnel (i.e., teachers and administrators); (b) reliability, validity, 
timeliness, and relevance of data; (c) use of findings for decision-making purposes; and 
(d) infrastructure development for sustainability after funding ends. 

In the first year of the project (2010-201 1), McREL designed and implemented a 
comprehensive evaluation protocol that employs a mixed-methods approach with process and 
outcome components that capitalizes on multiple sources of information and provides for the 
triangulation of findings. The evaluation incorporates data from a variety of collection strategies 
such as surveys, interviews, and reviews of extant documents and student education data. These 
data collection procedures continued, with modifications, in the second, third, fourth, and fifth years 
of the project (2011-2012, 2012-2013, 2013-2014, and 2014-2015). 

This report presents the evaluation findings from the fifth and final year of the project along 
with the findings from Years 1 through 4 when appropriate. Initial evaluation efforts emphasized 
the project’s implementation and short-term outcomes in order to provide opportunities for project 
improvement and feedback on the progress being made toward project goals. As the project 
progressed and data for effectively demonstrating the project’s impact have become available, the 
annual reports have increased the emphasis on the outcome evaluation. 

Process Evaluation 

The primary aims of the process evaluation are to provide feedback to JCPS and the targeted 
schools to enable quality improvement, identify successes and challenges, document the structures 
and processes related to project implementation, and adjust evaluation protocols based on 
experience and stakeholder input. The research questions for the process evaluation include the 
following: 

1. To what extent are the intervention components being implemented as planned (i.e., 
prototyping plans and progress on goals/ objectives) and with fidelity (i.e., consistent with 
evidence-based practice)? What is the status of reaching JCPS’ goals and objectives? 

2. What are the adaptations or deviations from the original plan, why, and what are the 
antecedents and consequences? How can barriers be minimized and best practices shared? 

3. Are interim student outcomes related to contextual, student, and implementation 
characteristics and activities? How do these data inform rapid prototyping and 
improvement? 

4. How do teacher-to-teacher relationships change within and across content areas and relate 
to interim student outcomes? 
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Specifically, the process evaluation includes a multilevel (i.e., school, teacher, and student) 
mixed-method analysis of the primary strategies of reform specified in the i3 application and revised 
by JCPS over time. Examples include the use of time for acceleration, immersion, and remediation; 
CAT; professional learning communities (PLCs); and master scheduling. Multiple methodologies 
are used to collect a range of proximal indicators that have been shown to correlate positively to key 
long-term educational outcomes such as student achievement and college readiness. Data sources 
include interviews, surveys, and extant data. 

Outcome Evaluation 

The outcome evaluation examines the effectiveness of the intervention components or 
strategies that impact proximal, intermediate, and distal outcomes for students, such as raising 
achievement for all students and narrowing achievement gaps. It also highlights any disparities in 
outcomes as well as best practices. The primary outcome research questions include the following: 

1 . What are the average effects on key school intermediate and long-term outcomes 

(e.g., pass rates, achievement rates, college readiness, and postsecondary enrollment rates)? 

2. What differential outcomes are experienced by students in the target schools? Do 
discrepancies relate to contextual, student, and implementation characteristics and 
activities? 

3. Is a best practice model identifiable that is scalable to other settings? 

Since the targeted schools were predetermined (i.e., categorized as persistently low- 
performing schools) and the intervention is schoolwide, the research design is non-experimental and 
relies on econometric modeling (Heckman, 2008). The outcome evaluation relies on the data 
collected in the process evaluation as well as data collected by the district (e.g., achievement rates, 
school climate perceptions by multiple respondents, and postsecondary enrollment rates). Core 
course pass rates are used as proximal achievement indicators; long-term achievement and college 
readiness are operationalized as students’ performance on the ACT®-related tests for English, math, 
science, and reading, including the ACT PLAN®' (hereafter referred to as PLAN) for 10th graders 
and the ACT for 11th graders. These data from national tests provide more stable measures over 
multiple years than the state’s high-stakes accountability test, which has undergone multiple revisions 
during this project period. 

Data Collection 

As mentioned previously, the evaluation utilizes multiple sources of data and incorporates a 
variety of stakeholder perspectives. Data are collected using teacher and student surveys; semi- 
structured interviews with principals. College Access Resource Teachers (CARTs), and school 
counselors; and secondary analyses of district data (e.g., student demographics, grades, and ACT and 
PLAN data). The following sections provide detailed information about the data collection methods 
that are part of the evaluation. 
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Interviews 


Although all principals, CARTs, and counselors were asked to participate in phone 
interviews between April and June 2015, response rates were low during Year 5 of the grant. Only 
one principal out of six (17%) and one CART out of six (17%) completed a phone interview. 
Response rates from counselors were slighdy better with three counselors responding (50%). 
Additionally, the project director was interviewed in the summer of 2015 to gather the perspective 
of the overall project from district leadership. 

Protocols for these 30- to 45-minute interviews consisted of structured and semi-structured 
questions. Principal interview protocols addressed all three areas of the grant (i.e., scheduling, 
advisory, and PLCs) and the progress that was made on the grant interventions over the course of 
the year while the CART protocols focused on how the teachers had implemented and sustained 
CAT advisory periods throughout the school year. Master scheduler/ counselor protocols, which 
were approximately 30- to 45-minutes, consisted of structured and semi-structured questions. For 
the complete interview protocols, refer to Appendix A. 

Surveys 

Multiple surveys were given throughout the 2014-2015 school year. JCPS administered the 
Comprehensive School Survey (CSS) to teachers and students in the spring of 2015. Additionally, 
separate surveys developed by the evaluation team were administered to principals, CARTs, 
teachers, and students. 4 The following sections provides details about each of the administered 
surveys. 


Principal and CART Implementation Surveys. In spring 2015, principals and CARTs 
were each asked to complete and submit their responses to an online survey via Qualtrics regarding 
implementation practices at their respective schools. The purpose of the surveys was to capture 
fidelity information regarding how the interventions were being implemented at each school. Three 
principals and three CARTs completed the surveys. See Appendix B for a copy of the surveys. 

Comprehensive School Survey (CSS). As mandated by the Jefferson County Board of 
Education, JCPS administers a districtwide CSS annually. The CSS assesses school characteristics 
and multiple dimensions of school climate with established reliability and validity. Students, 
teachers, and parents are asked to complete parallel versions of the survey each spring. Survey 
administration is overseen at the school level by a coordinator appointed by the principal. 

It should be noted that in past years, JCPS had used the Comprehensive School Climate 
Inventory (CSCI) developed by the National School Climate Center to measure indicators of school 
climate and culture. In Year 5 of the grant, JCPS decided to not pursue the CSCI in order to reduce 
costs and because the student- version of the CSS covered similar measures. To align the two 
instruments, evaluators conducted a crosswalk between the CSCI items and the CSS items. Data on 


4 The Student Survey replaced the student interviews in Year 3 in order to better capture perceptions from the full student 
population. 
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the CSS items that were similar to CSCI items were requested from JCPS. As shown in Table 2, 
across the i3 schools, the response rates for the student version of the CSS varied considerably, 
ranging from 64% to 94% with a mean of 79%. 


Table 2. Student CSS Response Rates Across the i3 Schools in Year 5 (2014-2015) 


School 

Student 
Enrollment a 

# of Surveys 
Completed 

Response Rate 

Academy @ Shawnee 

504 

351 

69.6% 

Fern Creek Traditional High School 

1,546 

1,381 

89.3% 

Moore Traditional High School 

1,028 

806 

78.4% 

Valley Traditional High School 

1,100 

871 

79.2% 

Waggener High School788 

766 

718 

93.7% 

Western High School 

810 

518 

64.0% 

Total Across i3 Schools 

5,745 

4,645 

79.0% 


In Year 5, a total of 4,645 students responded to the student CSS. Of those, 53% were male; 
46% were White; 41% were Black; 8% were Hispanic; 3% were Asian; 68% were students with free 
meal status; 7% were students with reduced-price meal status; 26% were students with paid meal 
status; and 4% were students with limited English proficiency (LEP). Additionally, 29% of the 
students were in the ninth grade, 26% were in the 10th grade, 25% were in the 11th grade, and 
20% were in the 12th grade. 

The CSS for School-Based Certified Staff Employees was used in Years 3, 4, and 5 to 
measure teachers’ perceptions of their ability to identify and appropriately respond to students’ 
unique academic and social needs. Items from this survey were also used to replace the CSCI during 
Year 5. Table 3 presents the Year 5 response rates across the i3 schools. 


Table 3. School Personnel CSS Response Rates Across the i3 Schools in Year 5 (2014-2015) 


School 

#of 

Professional Staff a 

# of Surveys 
Completed 

Response Rate 

Academy @ Shawnee 

53 

48 

90.6% 

Fern Creek Traditional High School 

95 

87 

9 1 .6% 

Moore Traditional High School 

136 

125 

9 1 .9% 

Valley Traditional High School 

1 12 

1 15 

I00.0% a 

Waggener High School 

67 

70 

I00.0% a 

Western High School 

62 

47 

75.8% 

Total Across i3 Schools 

525 

492 

91.6% 


Note. School personnel data were obtained from the JCPS website 
(http://www.jefferson.kl 2. ky.us/departments/acctresplan/Profiles/index.html) 

a When the number of surveys received was greater than the total number of school personnel counted, 1 00% response rate 
was reported. 
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As shown in Table 3, the response rate for the school personnel survey ranged from 76% to 
1 00% with a mean of 92%. School personnel respondents were also asked to provide demographic 
information. Of those with valid survey responses, 58% were female; 74% were White; 42% had at 
less than one year of teaching experience; and 20%, 22%, 7%, and 9% had two, three, four, and five 
years of teaching experience, respectively. 

Student Survey. During Years 1 and 2 of the project, student involvement (i.e., student 
voice) was captured through student focus groups. However, the evaluation and project team 
decided that a small subsample of students interviewed at each school did not accurately represent 
the entire school population. Furthermore, evaluators were limited in their findings due to attrition. 
For Years 3 through 5, the evaluation team created a survey to capture students’ perceptions 
regarding the core courses (i.e., English, math, science, and social studies) and their self-efficacy as 
students. The survey also asked a number of questions regarding CAT. The student surveys were 
distributed by the CART s during advisory time at each school. The survey was estimated to take 
approximately 15 to 20 minutes to complete. Table 4 shows the response rates by school for the 
2015 Student Survey. See Appendix C for a copy of the survey. 


Table 4. Student Survey Response Rates Across Schools in Year 4 (2014-2015) 


School 

# of Students 

# of Surveys 
Completed 

Response Rate 

Academy @ Shawnee 

504 

0 

0.0% 

Fern Creek Traditional High School 

1,546 

874 

56.5% 

Moore Traditional High School 

1,028 

144 

14.0% 

Valley Traditional High School 

1,100 

498 

45.3% 

Waggener High School 

766 

41 1 

53.7% 

Western High School 

810 

403 

49.8% 

Total Across i3 Schools 

5,745 

2,330 

40.6% 


Teacher Survey. The 2015 Teacher Survey, which can be found in Appendix D, 
contained a series of questions to examine teacher collaboration related to teaching and instruction. 
The survey asked detailed questions about the schools’ implementation of the three i3 interventions 
(i.e., CAT, PLCs, and trimester scheduling) and school context. 

The online survey was administered in April 201 5 to core content teachers from each of the 
six i3 schools. In general, the survey took respondents about 20 minutes to complete. The survey 
was voluntary; participants were not required to respond to any question and could opt out of the 
survey at any point. 

Before analyses were conducted, survey data were screened for missing response patterns. 
To ensure scale validity, cases with more than 75% of items with missing responses were omitted. 
Final sample sizes for each scale analysis are reported in the Findings section of this report. For the 
analyses, descriptive statistics were calculated to describe teachers’ perceptions of the project 
interventions and school context. 
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Core content teachers from each of the six i3 schools were given the opportunity to 
participate in the 2015 Teacher Survey (« = 94). As shown in Table 5, across the i3 schools, the 
response rates for the survey varied, ranging from 1 6% to 24% with a mean of 1 8%. 


T able 5. T eacher Survey Response Rates Across the i3 Schools in Year 5 (20 1 4-20 1 5) 


School 

# of Core Content 
Teachers a 

# of Surveys 
Completed 

Response Rate 

Academy @ Shawnee 

53 

1 1 

20.8% 

Fern Creek Traditional High School 

95 

15 

1 5.8% 

Moore Traditional High School 

136 

18 

1 3.2% 

Valley Traditional High School 

1 12 

26 

23.2% 

Waggener High School 

67 

16 

23.9% 

Western High School 

62 

8 

1 2.9% 

Total Across i3 Schools 

525 

94 

17.9% 


Extant Data and Artifact Review 


In addition to the CSS data described previously, JCPS’ Accountability, Research, and 
Planning Department provided several administrative datasets to McREL evaluators containing 
student data. Student-level data included 2014-2015 demographic information, ACT and PLAN 
scores, and postsecondary enrollment and involvement. 

ACT and PLAN Data. Datasets containing students’ ACT and PLAN scores were 
provided by JCPS to McREL evaluators for analysis. The ACT and PLAN exams cover four 
content domains including English, mathematics, reading, and science. The college readiness 
benchmarks, established by ACT, are the minimum ACT and PLAN test scores required for 
students to have a high probability of success in credit-bearing college courses. PLAN scores range 
between 1 and 32 and the established benchmarks for English, math, reading, and science are 15, 19, 
17, and 21, respectively. ACT scores range between 1 and 36 and the benchmarks for English, 
math, reading, and science are 18, 22, 21, and 24, respectively. During the 2014-2015 school year, a 
total of 1,404 students in 10th grade and 1,318 students in 11th grade from the i3 project schools 
took the PLAN and ACT exams, respectively. 

In this report, evaluators examined the extent to which students in the i3 schools met ACT 
and PLAN benchmarks. State and district-level data and findings from the project reports in 
Years 1, 2, 3, and 4 (2010-2011, 2011-2012, 2012-2013, and 2013-2014) are included in this 
document to show i3 schools’ progress over time. In addition, school differences in relation to the 
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ACT and PLAN outcomes are investigated by using chi-square tests. 3 Cramer’s H 5 6 7 is reported as an 
indicator of effect size. 


Subgroup analyses (e.g., student demographic characteristics) were conducted to investigate 
whether the percentages of students meeting or exceeding the ACT and PLAN benchmarks differed 
by student attributes. Since students were nested within schools (i.e., within a multilevel structure), 
and the percentages of students meeting or exceeding the benchmarks seemed to differ by school 
(Nixon et al., 2012), a series of hierarchical generalized linear models and subsequent analyses were 
conducted. The goal was to examine the associations between student attributes (i.e., gender, race, 
FRPM status, and LEP status) and the percentages of students passing the benchmarks for each 
subject area. 


Core Course Pass Rates. With assistance from JCPS program staff, evaluators identified 
and grouped all of the core courses into 10 categories: 


1 . English 

2. Algebra 

3. Geometry 

4. Senior Math 

5. Life Science 


6. Earth/Space Science 

7. Physical Science 

8. Social Studies 

9. U.S. History 

10. World Civilization 


Within each core course domain, three types of classes were identified: (1) comprehensive; 

(2) honors; and (3) others (e.g.. Advanced Placement [AP], advanced, and dual credit). Additionally, 
the core courses are subsequently grouped into four subject domains: (1) English; (2) math; 

(3) science; and (4) social studies (see Appendix E for the core course classifications). For this 
report, the analyses were conducted at the subject and core course level. 

Postsecondary Involvement. JCPS agreed to provide data from the National Student 
Clearinghouse’s StudentTracker database to assess students’ postsecondary attendance. To create an 
end-of-project target, 2010 StudentTracker data served as the baseline. According to the 
2010 StudentTracker data, 44% of students who graduated in May 2010 from the i3 project schools 
were enrolled in postsecondary institutions. The district mean for high schools for the same period 
was 55%; hence, the end-of-project target for the student transition outcome was set at 55%. This 
report indicates to what extent each i3 project school has made progress on attaining the 55% end- 
of-project target. 


5 Base on chi-square cell distributions, pairwise chi-squares were conducted to determine school differences in percentages of 
students meeting or exceeding the benchmarks. 

6 To interpret Cramer’s V, Ferguson (2009) suggests that a value of 0.20 was the recommended minimum effect size 
representing a “practically” statistically significant effect for social science data; a value of 0.50 presents a moderate effect size; 
and a value of 0.80 presents a strong effect size. 

7 When between-school variations in student outcomes are evident, multilevel modeling is a preferred approach because it 
adjusts for these variations. 


Fidelity 


To measure the extent of implementation of the intervention components in the i3 schools 
and to meet Government Performance Results Act (GPRA) reporting requirements, the evaluation 
must include a measure of fidelity for the project. Based on guidance provided by Abt Associates, 
the national evaluator of the i3 grant program, the fidelity instrument must align with the project’s 
logic model and consist of the following: 

• one or more indicators for each key component that defines the construct if 
implemented as planned; 

• operational definitions of the indicators, evidence needed to assess implementation of 
the component, and data sources and a schedule of data collection; 

• construct-level score across the indicators; 

• construct-level score categorized into levels of implementation fidelity; and 

• a definition of threshold for “adequate implementation.” 

A fidelity measure was designed for each of the three components of the grant. The six 
project schools were scored individually on each measure. Once calculated, the scores were 
aggregated across the schools to calculate one project score for each component. The fidelity 
measure for each component is included in the findings section. 
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Findings: Project Implementation Progress 


Several objectives were established to monitor the progress toward reaching each goal of the 
Making Time for What Matters Most project. The next few sections describe indicators of progress 
made toward those objectives during the fifth year of the i3 grant (2014-2015). When appropriate, 
findings from the Year 1, 2, 3, and 4 reports are presented for comparison purposes. 

Project Goal I : Student Mastery of Academic Material 

The first goal of the project is to provide stmctures and supports to facilitate student 
mastery of academic material and successful completion of all core courses in one year or less. 

Table 6 outlines the objectives for Project Goal 1 and the yearly or end-of-project target, when 
appropriate. 


Table 6. Project Goal I Objectives, Performance Measures, and Targets 


■ 

Objective 

Performance 

Measure 

T arget 

A 

Develop assignment protocols for core courses such 
that students are provided with options of acceleration, 
remediation, credit recovery, and increased time for 
study based on individual students’ progress and needs 

NA 

NA 

B 

Improve ongoing monitoring of students’ progress in 
core courses to enable rapid response to students’ 
academic needs 

NA 

NA 

C 

Increase students’ perceptions of academic challenge 

CSS 

75% of 

students agree 

D 

Increase students’ positive perceptions of their own 
academic self-efficacy 

NA 

NA 




English Year 5 
Target=94.0% 

E 

Increase the number of students who successfully pass 
core courses in one year 

Percentage of 
students who pass 
core courses during 
the academic year 

Math Year 5 
Target=87.5% 

Science Year 5 
Target=88.4% 




Social Studies Year 5 
Target=90.6% 

F 

Increase students’ perceptions of support for learning 
from teachers 

CSCI Support for 
Learning Scale 8 

3.41 


8 This was the project goal for Years I through 4. In Year 5, the CSCI was no longer utilized. Instead, the CSS was used as a 
measure for this objective to continue to monitor progress on these indicators. Because it was the first year in which the CSS 
had been used as a performance measure for the objectives, no end-of-project target was set. 
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As noted previously in the logic model (see Figure 1), this goal is accomplished by schools 
assigning master schedulers to develop a trimester schedule that strategically offers courses to assist 
students in acquiring enough credits to graduate on time. As students are more likely to pass their 
core courses, students’ perceptions of academic challenge, their own academic self-efficacy, and 
support for learning from teachers should also increase. 

Additionally, the project is designed to improve procedures to determine how student 
progress is monitored in real time so that appropriate remediation or acceleration modifications can 
occur or students can be moved to more appropriate course assignments. For example, school 
counselors may need to make decisions about remedial course interventions for students at risk of 
failing core courses. 

Other project strategies for this goal include increasing students’ perceptions of academic 
challenge, their own academic self-efficacy, and support for learning from teachers, as measured by 
the CSS and Student Survey. The following sections summarize the progress that the i3 project 
schools made in reaching the objectives of Goal 1 throughout Year 5. 

Objective A: Core Courses Assignment Protocol and Objective B: Ongoing 
Monitoring of Student Course Progress 

Objectives A and B include developing assignment protocols for core courses so that 
students are provided options of acceleration, remediation, credit recovery, and increased time for 
study based on individual students’ progress and needs. In spring 2015, evaluators conducted 
interviews with three of the counselors at the i3 schools to learn how they have placed students in 
courses and created their master schedules. The counselor interview had three objectives: (1) to 
determine how school counselors assign students to courses, including how freshmen placement 
differs from upperclassmen placement; (2) to determine if the trimester schedule meets students’ 
needs in terms of course placement (e.g., appropriate remediation or acceleration); and (3) to 
determine whether student progress (for the purpose of class placement) is monitored throughout 
the year, and if so, how it is monitored. 

The Teacher Survey also asked teachers questions regarding their perceptions of the school’s 
course assignment process. Additionally, students were asked on their respective survey to 
comment on whether they thought opportunities were available to make up courses in which they 
had failed previously and whether opportunities were provided for expanded learning experiences. 
Principals were also asked a set of questions regarding student course placement and scheduling on 
the spring 2015 principal survey and follow-up interview. Additionally, project staff were asked to 
reflect on the end-of-project results and lessons learned. Findings are presented in this section. 

Changes to the Trimester System. A primary goal of the grant was to establish more 
time for learning through the development of the trimester system 9 . Instead of having six to seven 


9 The type of schedule a school has is determined by school leadership. During the 20 1 4-20 1 5 school year, one of the 
i3 schools chose to return to a semester schedule. It is anticipated that more of the i3 schools will likely return to a semester 
schedule during the 20 1 5-20 1 6 school year. 
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courses per day, students had five 70-minute courses. On a trimester system, the courses end every 
12 weeks so classes can repeat more times throughout the year. This allows students who failed or 
struggled within a class to repeat the course, which allows them another opportunity to pass it 
within the same school year. If the student needs the course to graduate, students can earn the 
needed credit and graduate on time. 

Due to changes in the district, however, the distinction between the trimester and semester 
schedules became less clear in most schools. JCPS adopted end-of-course exams in English and 
math so the majority of English and math classes were offered every trimester making them full-year 
classes. Another issue that counselors commented on regarding trimesters is that if a student takes 
the first part of a sequenced course during first trimester, the second portion may not be available 
until the third trimester. For example, the first part of a foreign language class may be offered 
during the first trimester, but the second part may not be offered again until the third trimester. 
Thus, students may have 12 weeks without any instruction in that particular content area making 
learning continuity difficult. 

Although trimesters did not work out as envisioned for students, project staff credit the 
trimesters with fostering the PLC intervention by allowing for “parallel scheduling” so teachers of 
common courses have planning time together. The project director used the example that when two 
Algebra II teachers have the same planning time, they can immediately review student diagnostics. 

As a team, the teachers can then help each other. One might say, “My kids got this and yours didn’t. 
I’ll take these five kids tomorrow, and you take these five kids from my room.” This way, the 
teachers can work with different students, and ideally, the students may learn the lesson when it is 
presented in a different way from another teacher. 

For project staff, trimesters made a difference throughout the district because they allowed 
staff to realize the value of time and how being flexible with time and scheduling can benefit both 
teachers and students. One project staff member said. 

Trimesters have been the vehicle that has moved us further along with professional learning, moved 

us further along with standards-based assessments, and it’s moved us further along with the concept 

of time and using time creatively to increase student achievement. 

Course Assignment Approaches and Monitoring. At the high school level, 
counselors are responsible for assigning incoming freshmen as well as upperclassmen to courses. In 
previous years, for incoming ninth graders, JCPS district staff were involved in this process by 
providing data to the schools to assist them in assigning incoming ninth graders to appropriate 
classes (e.g., AP, honors, or comprehensive -level courses). Flowever, JCPS received feedback from 
schools indicating that the counselors would prefer to schedule their own students so JCPS stopped 
providing the schools with the data. However, during the counselor interviews, one interviewee said 
that the district spreadsheets “were incredibly helpful,” and they missed the support provided by the 
district. 


When asked to explain their process for scheduling the ninth graders, the counselors all 
explained that they used data, such as student scores from EXPLORE and the Kentucky 
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Performance for Education Progress assessment, along with students’ grades. Typically, at least one 
counselor from the high school also goes to the middle schools to meet with the students and 
discuss the high school requirements. 

Each school determines how many counselors will visit the middle school, and how the 
counselors will meet with the middle school students. For example, one of the counselors 
commented that when the four counselors from the high school visited the middle schools, they met 
with large groups of middle school students so the ratio was approximately four counselors to 
200 students. In the future, they plan on meeting with students in smaller groups so the ratio will be 
closer to one counselor to 30 or so students. 

To schedule upperclassmen, the counselors used similar processes. One counselor explained, 

We go to classrooms and explain the graduation requirements. Then, we meet individually with 

students to discuss the classes they need and their career goals. We try to schedule electives and extra 

academic classes to align with their career goals. 

As far as monitoring students and making adjustments to their schedule, counselors said that 
the changes are made on a regular basis throughout the year. If a teacher thinks that a particular 
course is too easy or too difficult for a student, they may make a recommendation that a student be 
placed in a course that could be more or less challenging for students. Parents and students may 
also request their child be placed in another classroom. When this happens, counselors will review 
the student data to make sure that the data supports moving the student before a final decision can 
be made. 

An interviewed principal also commented that moving students up to more challenging 
courses is considered a “celebration” at the school. Although it does not happen frequently, when 
teachers make the recommendation and a student is recognized for his or her hard work, it is seen as 
a success for everyone. 

Once the schedule has been established and students have been placed in courses, it 
becomes the teachers’ responsibility to provide instruction to the students. Thus, the teachers were 
asked to report on their perceptions of the course assignment process on the 2015 Teacher Survey. 
The Course Assignment Process scale gauges teachers’ perceptions of the course assignment 
process, which is how students are assigned to particular classes at their school. Items on the 
Course Assignment Process scale were as follows: 

• My school has a course assignment process that results in the best placement for 
students. 

• My school has a clear process for identifying students early when they are struggling. 

• My school has effective practices in place to successfully intervene with students who are 
struggling in a course. 

• My school has effective practices in place to move students up to a higher level course 
when they have been successful. 
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The internal consistency reliability (Cronbach’s a) estimate of teachers’ perceptions of the 
Course Assignment Process scale was strong at 0.86. Respondents selected answer choices on a 4- 
point Likert scale (1 = Strongly disagree, 2 = Disagree, 3 = Moderately agree , and 4 = Strongly agree). The 
items were summed to generate a scale score. 

Across the core content teachers at JCPS i3 schools (» = 94), 59% of the respondents 
reported more positive than negative perceptions about the effectiveness of the course assignment 
process (M — 10.39, SD = 2.79). For data display purposes, item ratings were grouped into negative 
and positive response categories (i.e., those responses rated at a 1 or 2 were considered “negative” 
while those rated at a 3 or 4 were considered “positive”). Figure 2 displays the percentage average 
across the i3 schools of teachers with negative and positive responses for the Course Assignment 
Process scale. 


40.8% 59.2% 


■ Negative ■ Postive 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

Figure 2. Distribution of teacher responses for the Course Assignment Process scale. 

Although the majority of respondents (59%) reported positive views about the course 
assignment process within their schools, differences existed in ratings across the individual items 
making up the Course Assignment Process scale. On average, teachers had positive views about the 
following statements: “My school has effective practices in place to successfully intervene with 
students who are struggling in a course” (63%); “My school has effective practices in place to move 
students up to a higher level of course when they have been successful” (62%); and “My school has 
a clear process for identifying students early when they are struggling” (60%). Although over half 
(52%) had a positive response to the item, “My school has a course assignment process that results 
in the best placement for students,” 48% had negative responses to the item. These findings suggest 
that teachers believe their system effectively identifies and intervenes when students are struggling, 
but students may not be assigned to the correct course based on the process. 

Impact of Trimester Scheduling on Struggling and Advanced Students. When 
asked to comment on how the trimester affected struggling and advanced students, interviewed 
counselors indicated that they could not see a major impact on struggling students, and one shared 
that the trimester system was actually a disadvantage for advanced students. A counselor said that 
the trimester schedule did not help students in preparing to take the ACT exam and that the 
upperclassmen were primarily taking year-long classes. This counselor also added that she feared 
some teachers may not be able to make adequate lesson plans for a 70-minute period. This 
counselor explained, however, that the trimester schedule has allowed for “flexibility and change” 
because it “allows us to meet students’ needs on an immediate basis.” Although, because of that 
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flexibility, she added that the master schedule changed every trimester, which caused many logistical 
challenges. The other counselors concurred with those statements. 

One counselor also stated that because of the schedule changes, a student could end up with 
three different teachers for the same subject in one year, which again, causes inconsistency for the 
students. For struggling students, a counselor said that she thought the opportunity to retake 
courses multiple times actually increased students’ opportunities to fail. Their expectation is now 
that they have multiple times to pass the class, they do not need to work as hard. While the reverse 
seems to be tme for advanced students as one counselor said that these students were “penalized” 
by the trimester schedule because they have fewer opportunities for electives or to take classes from 
the career pathway courses offered by the school. Another counselor said the trimester was 
“irrelevant” for advanced students because they were in year-long courses. 

In terms of the graduation rate, one counselor said that the trimester allowed students to 
earn additional credits, which they may not have had the opportunity to earn before, but there were 
many interventions happening at the school so changes in the graduation rate could not be 
attributed solely to the trimester schedule. Similarly, another counselor explained that everything 
changed once end-of-course exams were implemented, and the exams had more of an impact on the 
graduation rate than the schedule. 

Similarly, teachers on the 2015 Teacher Survey were asked if they thought that the trimester 
schedule addressed the needs of struggling students in their school. Of the 87 respondents, 
teachers’ perceptions were that the trimester schedule was not meeting the needs of struggling 
students with 61% (n — 34) indicating that it did not address the needs of those students while 39% 
(» =53) reported that it did meet their needs. 

Students were also asked questions on the 2015 Student Survey about the extent to which 
they believed they had opportunities to change courses if they were struggling and whether they 
could make up credits for courses they may not pass. Over three quarters (82%) of the students 
agreed to some extent that there were opportunities available to them to make up credits for courses 
that they do not pass. Just over half (56%) agreed that they are able to switch to different classes if 
they are struggling. Thus, it seems that the majority of students perceive that their schools are 
providing support for them if they are struggling. Table 7 provides student responses to these items. 


Table 7. Student Perceptions of Support if They are Struggling 


Items 

Disagree 
a Lot 

Disagree 
a Little 

Agree 
a Little 

Agree 
a Lot 

In general, if 1 am struggling in class, 1 have the 
opportunity to change to a different course. 

1 7.0% 

27.1% 

43.5% 

12.3% 

If 1 do not pass a class, there are opportunities for me to 

5.1% 

1 2.6% 

5 1 .0% 

31.3% 

make up the credit for that class. 
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Teachers on the 2015 Teacher Survey were asked if they thought that the trimester schedule 
addressed the needs of advanced or accelerated students in their school. Of the 87 respondents, 
63% (» = 55) indicated that the trimester schedule did not address the needs of advanced or 
accelerated students while 37% {n — 32) reported that it did meet their needs. 

Students were also asked questions on the 2015 Student Survey regarding opportunities to 
take classes outside their core courses (e.g., foreign languages, music, art, and theater) as well as 
opportunities to take honors or advanced classes. The majority (81%) of the student respondents 
agreed either a little or a lot that opportunities exist to take classes outside of the core courses and 
80% agreed a little or a lot that they have the ability to take honors or advanced courses. Thus, it 
seems that students perceive that their schools offer expanded and accelerated opportunities for 
them. Table 8 shows student responses to these items. 


Table 8. Student Perceptions of Expanded or Accelerated Learning Opportunities 


In general, . . . 

Disagree 
a Lot 

Disagree 
a Little 

Agree 
a Little 

Agree 
a Lot 

1 have opportunities to take classes outside my core 
classes (e.g., foreign language, music, art, and theater). 

5.7% 

1 3.8% 

46.6% 

33.9% 

1 have opportunities to take honors or advanced classes. 

7.4% 

1 2.7% 

40.7% 

39.2% 


Principals were asked on the implementation survey to describe success factors and 
challenges that have helped address the needs of all students in their schools, which are listed in 
Table 9. The successes and challenges were similar across the i3 schools. Collaboration, including 
PLCs and time, were listed as success factors as well as extended learning opportunities and 
standards-based grading. In regards to challenges, most of the comments related to the student 
body: students who are below grade level, truancy issues, and student transiency. One comment 
related to teacher experience and efficacy with the principal sharing that many staff members were 
nontenured and had alternative certifications. 


Table 9. Success Factors and Challenges in Addressing the Needs of All Students 


Success Factors 

Challenges 

PLCs 

Different levels of students make it challenging to 
teach and reach all students 

Time for teachers to plan 

Students [who are] multiple grade levels beneath 
[the] standards 

Weekly formative assessment data to assist with 
progress monitoring and interventions 

Truancy 

Collaborative efforts 

Students are transient; about 250 students come 
and go each year. 

Extended learning opportunities 

Teachers’ experience level and efficacy due to many 
nontenured staff members and alternative certifications. 

Hired interventionists 
Professional development for teachers 
Standards-based grading 
Tier II instructions 
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Objective C: Student Perceptions of Academic Challenge 

Student perceptions of academic challenge were assessed by the CSS and the 201 5 Student 
Survey. Additionally, teachers were asked to respond to questions regarding students’ academic 
challenge and academic engagement on the 2015 Teacher Survey. Academic challenge is defined as 
how easy or difficult a course is perceived by students and how motivated teachers think their 
students are to succeed. 

CSS Index on Student Perceptions of Academic Challenge. The evaluators and 
JCPS research staff selected an index consisting of three items from the student CSS to serve as an 
indicator of student perceptions of academic challenge. Table 10 shows the percentages of students 
who agreed or strongly agreed with the index items. Overall, about 67% of students agreed that 
they were academically challenged, which was about a 2% decrease from Year 4, and was 8% below 
the end-of-project target (75%). 

At the item level, findings from the Year 5 data were consistent with the findings from 
previous years. The item, “My teachers provide academically challenging content” received the 
highest levels of agreement (81%), followed by “Teachers at my school assign meaningful 
homework on a regular basis” (65%), and “I think school is fun and challenging” (56%). The item, 
“I think school is fun and challenging” received the lowest rating in Year 5 as compared to previous 
years. 


Table 10. Student Responses ( Agree or Strongly Agree) on the Academic Challenge Item Index 




School Mean 


CSS Academic Challenge Items 

Year 

Year 

Year 

Year 

Year 


1 

2 

3 

4 

5 

1 think school is fun and challenging. 

56.9% 

59.4% 

60.8% 

54.9% 

56.1% 

My teachers provide academically challenging content. 

76.0% 

77.4% 

78.8% 

78.4% 

80.6% 

Teachers at my school assign meaningful homework 
on a regular basis. 

66.1% 

61.5% 

65.7% 

62.7% 

64.8% 

Mean % of Agreement 


66.1% 


65.3% 

67.1% 


Figure 3 shows the mean of the percentages of students who agreed or strongly agreed with 
the item index across all six i3 schools. Overall, all of the project schools were still below the end- 
of-project target by the end of Year 5. As compared to previous years, the progress made from Year 
2 to Year 3 disappeared in Year 4. This decrease was largely contributed by the significant drop 
observed within Western High School between Years 3 and 4 (21% decrease). 
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Mean 

Academy @ 
Shawnee 

Fern Creek 

Moore 

Valley 

Waggener 

Western 

■ Year 1 

66.3 

66.5 

63.4 

69.6 

63.6 

62.0 

77.1 

■ Year 2 

66.1 

59.5 

64.9 

59.8 

58.4 

64.1 

86.9 

■ Year 3 

68.4 

60.4 

65.6 

64.8 

62.7 

67.7 

89.3 

■ Year 4 

65.3 

65.6 

67.1 

58.5 

67.1 

65.3 

68.2 

■ Year 5 

67.1 

58.5 

71.5 

63.3 

69.4 

65.8 

65.9 


Figure 3. Mean percentage of student perceptions of the 
Academic Challenge Item Index by school. 

Table 1 1 shows the mean of the index ratings of students’ perceptions of academic 
challenge. The analysis of mean differences by using a one-way ANOVA confirmed that schools 
did differ in ratings on student perceptions of the Academic Challenge Item Index, F( 5, 4637) = 
13.16,/) < 0.001, v] 2 = 0.01. 111 Post hoc comparisons were conducted to compare the mean ratings 
across all schools. Fern Creek Traditional High School, Valley Traditional High School, Waggener 
High School, and Western High School had a similar level of academic challenge reported, while 
Academy @ Shawnee had a significandy lower mean rating as compared to all of the other i3 
schools, except Moore Traditional High School (p < 0.05). The means and standard deviations at 
the item-level for each i3 school are reported in Appendix G. 


Table I I. Mean Ratings of Student Perceptions of the Academic Challenge Item Index 
by School 


School 

n 

M 

SD 

Academy @ Shawnee 

351 

2.56 

0.63 

Fern Creek Traditional High School 

1,380 

2.76 

0.52 

Moore Traditional High School 

805 

2.63 

0.60 

Valley Traditional High School 

871 

2.78 

0.60 

Waggener High School 

718 

2.70 

0.57 

Western High School 

518 

2.68 

0.54 

Across All i3 Schools 

4,643 

2.71 

0.57 


10 To interpret r) 2 , Ferguson (2009) suggests that a value of 0.04 was the recommended minimum effect size representing a 
“practically” significant effect for social science data; a value of 0.25 presents a moderate effect size; and a value of 0.64 presents 
a strong effect size. 
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Subgroup analyses using one-way ANOVAs and independent sample /-tests were conducted 
to examine whether the mean ratings of student perceptions of the Academic Challenge Item Index 
differed across the students’ demographic characteristics. Specifically, evaluators examined 
differences in perceptions by gender, LEP 11 status, grade level, FRPM status, and race/ ethnicity. 
Several demographic variables moderated students’ perceptions of academic challenge. Post hoc 
comparisons were conducted when appropriate. In summary, the analyses revealed the following 
results: 

• Male students and female students did not differ in their perception of academic 
challenge, /( 4641) = -0.387, /> = 0.698, Cohen’s d= 0.00. 12 

• LEP students perceived a higher level of academic challenge than did non-LEP students, 
<4641) = -5.895,/) < 0.001, Cohen’s d= 0.52. 

• Students from higher grade levels perceived a higher level of academic challenge than did 
students from lower grade levels, F( 3, 4626) = 5.97,/) < 0.001, rp = 0.004. Specifically, 
12th graders reported a higher level of academic challenge than did 10th graders 

(p — 0.032); and 11th graders reported a higher level of academic challenge than did 
ninth and 10th graders (p = 0.011 and p = 0.002, respectively). 

• Student perceptions of academic challenge did not differ by their FRPM status, 

F( 2, 4640) = 0.37 ,p = 0.690, rf = 0.000. 

• Student perceptions of academic challenge did differ by students’ race/ ethnicity, 

F(4, 4638) = 9.82,/) < 0.001, rf = 0.008. Specifically, Asian students reported higher 
levels of academic challenge than did students from other racial/ ethnic backgrounds, 
including White, Black, and Hispanic students, as well as students categorized as having 
“other” racial/ ethnic backgrounds (p < 0.001,/) < 0.001,/) = 0.005,/) < 0.001, 
respectively). Additionally, Hispanic students reported a higher level of academic 
challenge than did White students (p = 0.026). 

Table 12 shows the changes in students’ perceptions of academic challenge over time by 
subgroups. From Year 1 to Year 5, the mean increased for females; students receiving reduced-price 
meals and students with paid meal status; White and Asian students; and 11th and 12th graders. 


1 1 JCPS’ policy is that any student who does not pass an English proficiency test is considered a LEP student, which includes 
English as a Second Language students. 

12 Cohen’s d is a common effect size estimating the magnitude of the mean differences. In practice, 0.3 is a small effect size, 
0.5 is a medium effect size, and 0.8 is a large effect size (Tavakol & Dennick, 201 I). 
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Table 12. Changes in Student Perceptions of Academic Challenge Over Time by Subgroup 



On the 2015 Student Survey, students were asked to reflect on the extent to which their core 
courses (i.e., English, math, science, and social studies) motivated them to learn. Figure 4 shows 
how the core courses compared to each other and how student responses differed from Years 3 
and 4. In Year 5, nearly three quarters of the students (74%) agreed that English motivated them to 
learn more about the subject, which increased from 64% in Year 3 and 65% in Year 4. Math and 
science were fairly similar in the percentage of students who were motivated to learn more about 
those subjects (54%). Although math decreased by about 4%, science stayed relatively stable. 
Elowever, less than half the students (47%) agreed that social studies motivated them to learn more, 
but the percentage remained relatively stable from Year 4 and still showed an increase from Year 3 
(45%). 
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Figure 4. Extent to which the core content courses motivate students 
to learn more about the subject 

To learn more about which courses were found to be difficult or challenging, students were 
asked how much they agreed or disagreed that the core content areas were too hard for them. 

Figure 5 shows students’ level of agreement that their core content courses were too hard in Years 3, 
4, and 5. Across all of the content areas, the percentage of students reporting that the classes were 
too hard increased from Year 4 to Year 5. Approximately a fifth of the students reported that 
English (21%), science (22%), and social studies (20%) were too hard while 32% reported that math 
was too hard. 

100% 


75% 


50% 


25% 


0% 


29.3% 31 - 7% 29.3% 



English Math Science Social Studies 


■ Yr 3 ■ Yr 4 «Yr5 

Figure 5. Extent to which the core content courses are perceived 
as being too hard by students 
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Teacher Perceptions of Student Academic Challenge. The Student Academic 
Challenge scale on the 2015 Teacher Survey consisted of six items that measured teachers’ 
perceptions of the extent that students are challenged in their schools. Items in this section include: 

• In general, students seem to enjoy my school. 

• In general, high grades at my school are viewed as an indicator of easy course work. 

• In general, students seem to be bored at my school. 

• In general, students at my school perceive challenge as a way to go beyond expectations. 

• In general, high grades at my school are viewed as an indicator of mastering high 
standards. 

• In general, students do well in my school. 

The internal consistency reliability (Cronbach’s a) estimate of the Student Academic 
Challenge scale was 0.53. Respondents selected answer choices on a 4-point Likert scale 
(1 = Strongly disagree , 2 = Disagree, 3 = Agree, and 4 = Strongly agree). 

Several items on the Student Academic Challenge scale tended to have high teacher 
perception ratings, including “In general, students seem to enjoy my school” (69%) and “High 
grades at my school are viewed as an indicator of mastering high standards” (76%). Furthermore, 
nearly three-quarters of the teacher respondents disagreed or strongly disagreed with the statement, 
“High grades at my school are viewed as an indicator of easy course work” (67%). This indicates 
that teachers perceive the courses as rigorous and that grades are not a reflection of coursework 
which has been designed to be easy for students. 

However, over a third (36%) of the teachers agreed that “In general, students seem to be 
bored at my school.” Similarly, only 34% of the teachers responded that “Students at my school 
perceive challenge as a way to go beyond expectations.” Interestingly, the teachers were split when 
asked if students do well at their schools with just under half (44%) disagreeing to some extent with 
that statement while 56% agreed to some extent (see Table 13 for teachers’ perceptions of student 
academic challenge). 


Table 1 3. Teacher Perceptions of Student Academic Challenge by Item 


In general, . . . 

N 

M 

SD 

students seem to enjoy my school. 

86 

2.72 

0.64 

high grades at my school are viewed as an indicator of easy course work. 

86 

2.34 

0.59 

students seem to be bored at my school. 

86 

2.38 

0.58 

students at my school perceive challenge as a way to go beyond expectations. 

86 

2.24 

0.72 

high grades at my school are viewed as an indicator of mastering high standards. 

86 

2.84 

0.67 

students do well in my school. 

86 

2.55 

0.61 
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Minor variations also existed among the i3 project schools (see Table 14 to see how the 
schools compared to one another on the Student Academic Challenge scale). Because of the small 
sample size for each school, these findings should be interpreted with caution, but subgroup 
analyses reveal that teachers’ perceptions of student academic challenge differed significantly by 
school. Post hoc comparisons suggested statistically significant differences between Fern Creek and 
Moore Traditional High Schools (p — 0.010) and Waggener and Moore Traditional High Schools 
(p = 0.022). Specifically, Fern Creek (M = 16.38, SD = 3.07) and Waggener (M = 15.80, SD — 1.01) 
Traditional High Schools reported higher ratings than Moore Traditional High School (M = 13.75, 
SD = 1.61). 

Table 14. Teacher Perceptions of Student Academic Challenge by School 


School 

N 

M 

SD 

Academy @ Shawnee 

10 

15.10 

1.91 

Fern Creek Traditional High School 

13 

16.38 

3.07 

Moore Traditional High School 

16 

13.75 

1.61 

Valley Traditional High School 

24 

14.63 

1.81 

Waggener High School 

15 

15.80 

1.01 

Western High School 

8 

15.50 

2.07 

Across All i3 Schools 

86 

15.19 

1.58 


Teacher Perceptions of Student Academic Engagement. The Student Academic 
Engagement scale, consisting of six items, measures teachers’ perceptions of the extent that students 
are academically engaged in their school. Items included in this section are: 

• Teachers at my school challenge students academically. 

• Teachers at my school encourage students to go beyond stated expectations. 

• Teachers at my school involve students in setting expectations. 

• Teachers at my school involve students in planning lessons. 

• Teachers at my school involve students in developing criteria for assessing their 

assignments. 

• Teachers at my school work with disenfranchised students to help them feel more 
connected to school. 

The internal consistency reliability (Cronbach’s a) estimate of the Student Academic 
Engagement Scale was 0.81. Respondents selected answer choices based on a 4-point Likert scale 
(1 = Strongly disagree , 2 = Disagree, 3 = Agree, and 4 = Strongly agree) with a summed scale range of six 
to 24 points. 

Core content teachers at JCPS i3 schools ( n — 86), on average, responded more positively 
than negatively about student academic engagement (M — 16.31, SD = 2.72). For data display 
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purposes, item ratings were grouped into negative and positive response categories (i.e., those 
responses rated at a 1 or 2 on the Likert scale were considered “negative” while those rated at a 3 or 
4 were considered “positive”). Figure 6 displays the percentage average across the i3 schools of 
teachers who had negative or positive responses on the Student Academic Engagement scale. 


16 . 4 % 83 . 6 % 


■ Negative ■ Positive 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

Figure 6. Distribution of teacher responses for the Student Academic Engagement scale 

Overall, nearly two thirds (63%) of the respondents demonstrated positive agreement on the 
Student Academic Engagement scale. The percentage of teachers responding with a rating of agree 
or strongly agree for students’ academic engagement is more than 80% on two items: “Teachers at my 
school challenge students academically” (81%) and “Teachers at my school work with 
disenfranchised students to help them feel more connected to school” (86%). Additionally, nearly 
80% of teachers agreed with the item, “Teachers at my school encourage students to go beyond 
stated expectations” (79%) and over 60% agreed with the item “Teachers at my school involve 
students in setting expectations” (65%). 

For the other two items, teachers’ perceptions of students’ academic engagement tended to 
be more negative. For example, 56% of teachers disagreed with the item, “Teachers at my school 
involve students in developing criteria for assessing their assignments,” while nearly three quarters 
(76%) disagreed with the statement, “Teachers at my school work to involve students in planning 
lessons.” 


Minor variations also existed among the i3 project schools (see Table 15 to compare schools 
on the Student Academic Engagement scale). Because of the small sample size for each school, 
these findings should be interpreted with caution, but subgroup analyses reveal that teachers’ 
perceptions of students’ academic engagement differed significantly by school. Post hoc 
comparisons suggested statistically significant differences between the Academy @ Shawnee and 
Moore Traditional High School (p = 0.046) and Valley Traditional High School (p — 0.026). 
Specifically, the Academy @ Shawnee (M = 17.50, SD = 2.07) reported higher ratings than Moore 
Traditional High School (M = 15.38, SD = 3.18) and Valley Traditional High School (M = 15.29, 
SD = 2.53). 
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Table 1 5. Teacher Perceptions of Student Academic Engagement by School 


School 

N 

M 

SD 

Academy @ Shawnee 

10 

17.50 

2.07 

Fern Creek Traditional High School 

13 

17.38 

3.12 

Moore Traditional High School 

16 

15.38 

3.18 

Valley Traditional High School 

24 

15.29 

2.53 

Waggener High School 

15 

16.93 

1.91 

Western High School 

8 

16.88 

2.64 

Across All i3 Schools 

86 

16.56 

2.58 


Objective D: Student Perceptions of Academic Self-Efficacy 

Objective D seeks to increase students’ positive perceptions of their academic self-efficacy. 
This was measured on the 2015 Student Survey by asking students questions about how well they 
are doing in school. The majority of students (88%) reported that they are good students, while 
28% responded that they were not doing well in school. Therefore, some students indicated that 
while they perceived themselves to be good students, they were not doing well in school. This could 
be due to a number of factors, including interpretations of what being a “good student” means (i.e., 
staying out of trouble or getting good grades), or students may try hard but still find that they are 
not doing well. 

To examine student perceptions more deeply, a series of items were asked regarding their 
academic self-efficacy. Table 16 shows the extent to which students were sure “things would work 
out” on the items related to this constmct. Less than a fifth of the students were very sure about any 
of the items. However, over 70% were mostly or very sure on two of the items: (1) “When you have to 
learn something new at school” (80%), and (2) “when you have to do an activity for the first time 
(73%).” Students were less likely to be sure of items related to public speaking (39% were not sure at 
all or mostly not sure) or when they were having trouble with their school work (42% were not sure at all 
or mostly not sure). 


Table 16. Student Perceptions of Academic Self-Efficacy 


How sure are you that things will work out . . . 

Not Sure 
at All 

Mostly 
Not Sure 

Mostly 

Sure 

Very Sure 

when you have to learn something new at school? 

4.7% 

15.8% 

62.4% 

17.1% 

when you have to give a talk in class? 

1 3.3% 

26.1% 

45.1% 

1 5.4% 

when you have to do an activity for the first time? 

5.7% 

1 9.2% 

59.2% 

1 5.9% 

when you are having trouble with your school work? 

8.9% 

32.7% 

48.6% 

9.9% 

when you have to figure out something by yourself? 

7.0% 

25.4% 

53.1% 

14.5% 


28 





Objective E: Student Core Course Completion Rates 


As mentioned previously, all of the core courses were grouped into 10 categories: 


1 . English, 

2. Algebra, 

3. Geometry, 

4. Senior math, 

5. Life science, 


6. Earth/ space science, 

7. Physical science, 

8. Social studies, 

9. U.S. history, and 

10. World civilization 


Within each core course domain, three types of classes are identified: (1) Comprehensive, 

(2) honors, and (3) others (e.g., AP, advanced, and dual credit). Additionally, the core courses are 
subsequently grouped into four subject domains: (1) English, (2) math, (3) science, and (4) social 
studies (see Appendix E for a listing of the core course classifications). For this report, the analyses 
were conducted at the subject and core course level. The yearly project targets were defined by 
JCPS based on the Year 1 pass rates and are used to examine i3 schools’ status in terms of meeting 
project targets (see Table 17). 


Table 1 7. Project Yearly Course Passing Targets 


Subject 

Year 1 
Pass Rate 
(Baseline) 

Yearly 
Increase 
T argets 

Year 2 
T arget 

Year 3 
T arget 

Year 4 
T arget 

Year 5 
T arget 

English 

90.0% 

1 .0% 

9 1 .0% 

92.0% 

93.0% 

94.0% 

Math 

80.5% 

2.0% 

82.5% 

84.5% 

86.5% 

87.5% 

Science 

82.9% 

1.5% 

84.4% 

85.9% 

87.4% 

88.4% 

Social Studies 

85.1% 

1.5% 

86.6% 

88.1% 

89.6% 

90.6% 


With regard to the analysis, since students were nested within schools, a series of 
Generalized Linear Mixed Models (GLMM) were conducted to examine whether student pass rates 
varied across subgroups after considering school-level variation. Figure 7 and Table 18 shows the 
pass rates for subjects, core courses, and subcategories. The overall pass rate across the i3 schools 
was approximately 82%; the same as the past two years. Specifically, the pass rates for English, 
math, science, and social studies were 86%, 79%, 77%, and 84%, respectively. As compared to 
project’s yearly targets, i3 students did not reach the targeted levels in Year 5 (6% to 11% below the 
targets). As compared to Year 4 findings, i3 students showed one percentage point increase in 
English and social studies, but exhibited one and three percentage point decreases in math and 
science, respectively. 
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Figure 7. Longitudinal trend of student core course pass rates from Year I to Year 5 


When examining all 1 0 core courses, similar to what were found in previous years, students 
continued to struggle the most in their Geometry courses (73%, 75%, 75%, and 75% pass rates for 
Year 2, 3, 4, and 5, respectively). As compared to Year 4, students showed noticeable regression in 
their Life Science courses (a decrease of 6 percentage points). Across the core courses, similar to 
those reported in previous years, student pass rates were higher for students taking honors (80%) 
and other (94%) classes relative to students enrolled in comprehensive (79%) classes. These 
findings are not surprising as students enrolled in honors courses are more likely to pass their 
courses than students taking general courses. 


Table 18. Percentages of Students Passing Core Courses by Subcategories 




All 

Core Course Subcategory 

Subject 

Core Course 

N 

% 

Comprehensive 
n % 

Honors 
n % 

Other 
n % 

English 

Overall English 

6,936 

85.9% 

3,765 

83.0% 

1,797 

85.0% 

1,374 

95.3% 


Geometry 

1,757 

74.2% 

848 

65.0% 

562 

76.7% 

347 

92.5% 

Algebra 

2,828 

78.0% 

1,566 

73.4% 

961 

79.6% 

301 

96.7% 


Senior Math 

966 

92.0% 

450 

89.6% 

1 16 

96.6% 

400 

93.5% 

Overall Math 

5,551 

79.2% 
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All 


Core Course Subcategory 


Subject 

Core Course 


Comprehensive 

Honors 

Other 



N % 

n % 

n % 

n % 


Life Science 

2,647 

75.3% 

1,042 

73.1% 

961 

68.5% 

644 

88.8% 

Earth/Space Science 

1,61 1 

78.3% 

930 

71.4% 

580 

86.0% 

101 

98.0% 

Social 

Physical Science 

1,144 

8 1 .0% 

832 

77.2% 

133 

88.0% 

179 

93.9% 

Overall Science 

5,402 

77.4% 







Social Studies 

1,431 

78.8% 

848 

77.7% 

244 

66.8% 

339 

90.3% 

U.S. History 

1,271 

88.2% 

597 

85.6% 

373 

87.1% 

301 

94.7% 

Studies 

World Civilization 

1,472 

86.5% 

834 

86.6% 

387 

78.8% 

251 

98.4% 

OVERAl 

Overall 
Social Studies 

4,174 

84.4% 







_L 



1 1,712 

78.5% 



4,237 

93.8% 


Student Pass Rates by School 


The results of the chi-square tests 13 suggest that some schools had higher pass rates than 
others,^ (5) = 373.24,/) < 0.001, Somer’s d — 0.04. For instance, as shown in Table 19, Moore 
Traditional High School had the highest overall pass rate (88%), while Western High School had the 
lowest overall pass rate (72%). As compared to the Year 5 project targets, Moore Traditional High 
School was 0.6% shy from the target in science and was 1% above the target in social studies. The 
detailed descriptive data for each core course by school are reported in Appendix F. 


Table 19. Pass Rates by Subject Area by School 


Subject 

Academy 

@ 

Shawnee 

Fern 

Creek 

Moore 

Valley 

Waggener 

Western 

Year5 
T arget 


n 

% 

n 

% 

n 

% 

n 

% 

n 

% 

n 

% 


English 

686 

82.7% 

2071 

89.1% 

1,064 

88.9% 

1,080 

82.9% 

1,195 

84.2% 

840 

83.6% 

94.0% 

Math 

457 

79.0% 

1601 

82.8% 

1,068 

84.6% 

865 

84.0% 

825 

76.2% 

735 

61.5% 

87.5% 

Science 

538 

72.1% 

1518 

78.0% 

876 

87.8% 

693 

77.8% 

678 

80.8% 

901 

66.5% 

88.4% 

Social Studies 

392 

70.2% 

912 

86.4% 

783 

9 1 .6% 

772 

90.0% 

644 

83.7% 

509 

75.9% 

90.6% 

OVERALL 

2,073 

76.7% 

6,102 

84.3% 

3,791 

88.0% 

3,608 

83.4% 

3,342 

81.4% 

3,147 

71.9% 

-- 


*Results of chi-square tests were significant at the level of 0.05 across all subject areas with a small effect size (ES = 0.0 1 -0. 1 0). 
Detailed chi-square tests statistics are reported in Appendix F. 


Subgroup Analyses. Analyses were conducted for six subgroups: (1) trimester schedule 
status (i.e., two-term versus three-term courses); (2) gender; (3) LEP status; (4) grade level; 


13 Pearson Chi-Square tests are reported. Effect sizes were estimated based on Somers’ d (Ferguson, 2009): a value of 0.2 is 
small effect size; a value of 0.5 is moderate, and a value of 0.8 is large. 
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(5) FRPM status; and (6) race/ethnicity. Detailed statistical results, including coefficients (J3) and 
odds ratios (OR) 14 are reported in Appendix F. Key findings of the subgroup analyses are 
summarized in the following sections. 

Student pass rates were higher among students taking three-term courses than those 
taking two-term courses. The overall pass rate for students taking three-term courses and two- 
term courses was 82% and 96%, respectively. Results of the GLMM indicated that the difference in 
pass rates were statistically significant, /? = -1.26 ,p < 0.001, OR = 0.28. Similar findings are shown 
in all subject areas, except science. Specifically, the odds ratio of passing English, math, and social 
studies was 0.30, 0.08, and 0.20 times higher for students taking three-term courses (i.e., trimester) in 
comparison with students taking two-term courses (i.e., semester). Interestingly, although no 
difference was found with the overall science courses, differences were observed in the sub-subject 
areas for the science courses. For instance, the pass rate for Life Science was higher among the 
three-term courses (OR = 0.40) as compared to two-term courses; however, the pass rate for 
Earth/Space Science courses was higher among two-term courses (OR = 7.63) as compared to 
three-term courses. Taken together, trimester scheduling seemed to have overall positive effects in 
English, math, social studies, and a sub-area of the science courses (i.e.. Life Science), but had a 
reversed effect in another sub-area of science (i.e., Earth/Space Science). 

The pass rates were lower for males than for females. The overall pass rates for males 
and females were 78% and 86%, respectively. The GLMM results suggested that the difference in 
pass rates was statistically significant, /? = 0.53 ,p < 0.001, OR = 1.69. In other words, the odds of 
passing the courses was 1.69 times higher if the student was a female. This difference was consistent 
across all subjects and core courses, except Geometry and Senior Math, with the OR s ranging from 
1.17 to 1.99. 

There was no difference in pass rates across all subject and core courses between 
LEP and non-LEP students. The overall pass rate for students with LEP and without LEP was 
84% and 82%, respectively. The GLMM results showed that such a difference was not statistically 
significant. This finding is consistent across all subject and core courses. 

Twelfth graders had higher pass rates in all subject areas in comparison with ninth, 
10th, and 11th graders. The overall pass rate for ninth, 10th, 1 1th, and 12th graders was 79%, 

81%, 82%, and 90%, respectively. Using 12th-grade students as the reference group, GLMM results 
indicated that 12th graders had a higher overall pass rate than ninth, 10th, and 11th graders, with 
ORs ranging from 0.37 to 0.43. A similar pattern was found in all subject areas as well as the 
majority of the core courses with few exceptions. 

Students with free meal status had lower pass rates in all subject areas in comparison 
with students with paid meal status; there was no difference between students with reduced- 
price meal status and students with paid meal status. The overall pass rates for students with 
free, reduced-price, and paid meal status was 81%, 84%, and 88%, respectively. Specifically, results 


14 OR = the odds of passing the course was [odds ratio] times higher (or lower) if the student was a “non-reference group” in 
comparison with “the reference group”. 
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of GLMM indicated that the odds of passing core courses were 0.64 times lower for students with 
free meal status in comparison with students with paid meal status. Similar patterns were observed 
across all subject areas and the majority of the core course areas, with ORs ranging from 0.30 to 


0.72. 


Student pass rates varied by race. The overall pass rate for students who are White (or 
Caucasian), Black (or African American), Hispanic, Asian, as well as those categorized into an 
“other” group (e.g., American Indian, Pacific Islander, and multiracial) are 83%, 79%, 87%, 95%, 
and 85%, respectively. Specifically, the odds of passing courses was 0.76 times lower for Black 
students in comparison to White students. In contrast, the odds of passing courses was 3.34 and 
1.46 times higher for Asian and Hispanic students in comparison to White students. Examination 
of racial/ ethnic differences across all subject areas revealed similar patterns. In particular, 

• Compared to White students, Black students had lower pass rates in math (OR = 0.62) 
and science subjects (OR = 0.70). 

• Compared to White students, Hispanic students had higher pass rate in math, science, 
and social studies, with ORs ranging from 1.52 to 1.84. 

• Compared to White students, Asian students had higher pass rates in math 
(OR = 5.33) and social studies (OR = 13.38). 

• There were no differences between White students and students in the “other” 
racial/ethnic group across all core courses and subject areas. 

Objective F: Student Perceptions of Support for Learning from Teachers 

Objective F is designed to measure changes in students’ perceptions regarding the support 
for learning they have received from teachers. During Years 1 through 4, the CSCI Support for 
Learning scale was used as an indicator of students’ perceptions of teachers’ support for their 
learning. As explained previously, in Year 5, JCPS decided to not use the CSCI because the CSS 
collected similar information. Thus, McREL evaluators conducted a crosswalk of the CSCI and CSS 
items and requested that JCPS provide data on the CSS items that were similar to the CSCI items. 

One item, “My teachers respect my opinion in class even if it disagrees with their opinions,” 
from the student CSS was used as an indicator of students’ perceptions of teachers’ support of 
student learning. Of the valid student responses (n = 4,633), the mean of the item was 2.89 
(SD = 0.70). As illustrated in Figure 8, students reported feeling more positive than negative about 
the Support for Learning scale item. 
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My teachers respect my opinion in class even if it 
disagrees with their opinions 

■ Negative ■ Positive 

0% 20% 40% 60% 80% 100% 

Figure 8. Distribution of student responses for the Support for Learning scale. 

Note. Each CSS item was measured on a 4-point Likert scale (I = Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly 
agree), ratings of I and 2 were grouped as a negative response and ratings of 3 and 4 were grouped as a positive response. 

The examination of between-school differences in the Support for Learning subscale 
suggests that, in Year 5, schools varied in their levels of support for learning, F( 5, 4627) = 7.61, 
p < 0.001, rf = 0.008. In particular, students at Fern Creek Traditional High School perceived a 
significantly higher level of support for learning compared to students from other high schools, with 
the exception of Waggener High School {p < 0.01); students from Waggener High School perceived 
a significantly higher level of support for learning compared to students from Valley Traditional 
High School (p = 0.029). 

Subgroup analyses were conducted to assess whether students’ demographic characteristics 
moderated their responses on the Support for Learning subscale. The demographics included 
gender, grade level, race/ ethnicity (i.e., White, Black, Hispanic, Asian, and other racial/ ethnic 
groups), FRPM status, and LEP status. Findings are summarized as follows: 


20 . 2 % 79 . 8 % 


• Student ratings of teachers’ support for learning did not vary by gender (/(4631) — 0.76, 
p - 0.446, rf = 0.000). 

• Student ratings of teachers’ support for learning did not differ by FRPM status 
(F(2, 4630) = 1.33,/ = 0.264, rf = 0.001). 

• Students with LEP (M = 3.02, SD = 0.70) had higher ratings of teachers’ support for 
learning than students without LEP (M = 2.89, SD = 0.70). 

• Student ratings of teachers’ support for learning did differ by grade level, 

F( 3, 4617) = 4.27,/ = 0.005, yf = 0.003. Specifically, 12th graders reported a higher 
level of support for learning from their teachers than did ninth graders (p = 0.01 1). 

• Student ratings of teachers’ support for learning did differ by race, F( 4, 4628) = 3.65, 

p = 0.006, v] 2 = 0.003. In particular, Hispanic students reported a higher level of support 
for learning from their teachers than did White students (p = 0.048). 

Implementation Fidelity for Scheduling 

Each i3 project is required to develop an implementation fidelity rubric for the primary 
interventions, which has been approved by the national evaluation team. The fidelity score for 
Project Goal 1 is high as most of the indicators for this project goal have been mandated by JCPS. 
The indicators for this construct follow: 
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• Master scheduler assigned 

• Five 70-minute courses per day for each of the three 12-week trimesters 

• Students are assigned to courses based on data (e.g., academic history and assessment 
scores) 

Each project school was scored individually on these indicators. The scores were aggregated 
across the schools for one fidelity score. Table 20 shows the full fidelity table and scores. All six i3 
project schools received a score of 3 or higher, which indicates that they have implemented this 
component of the project with fidelity. 
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Table 20. JCPS Fidelity Measure and Scores for Component I: Student Scheduling 


Key Elements of 
Component 

Operational 
Definition 
for Indicator 

Data Source(s) 
for Measuring 
the Indicator 

Data 

Collection 
Schedule for 
the Indicator 

Raw Scoring and Scoring Criteria for Fidelit 

Possible 

Indicator 

Score Per Scoring School 

School Criteria Name 

Y 

Indicator 

Score 

1 

Master scheduler 
assigned to each 

Master scheduler 
has been assigned 

District and 
school records; 

Interviews 
conducted by the 

0-1 

0 = Low 

Master scheduler 

Academy @ 
Shawnee 

1 


school 


interviews with 
counselors; 
survey results 

evaluation team 
annually 


is not assigned 

Fern Creek 

1 





1 = High 

Master scheduler 

Moore 

1 







is assigned 

Valley 

1 








Waggener 

1 








Western 

1 

2 

Five 70-minute 
courses per day for 

Schedules are based 
on five 70-minute 

District and 
school records 

Records 
provided by the 

0-1 

0 = Low 

School schedules do not follow 

Academy @ 
Shawnee 

1 


each of the three 
1 2-week trimesters 

courses per day for 
each of the three 


district and 
collected by the 
evaluation team 
annually 


the five 70-minute courses per 
day schedule 

1 = High 

Fern Creek 

1 


1 2-week trimesters 



Moore 

1 






School schedules follow the five 
70-minute courses per day 
schedule 

Valley 

0 







Waggener 

1 








Western 

1 

3 

Students are 
assigned to courses 

Counselors have 
assigned students to 

Counselor 
interviews and 

Interviews 
conducted by the 

0-2 

0 = Low 

No students are assigned based 

Academy @ 
Shawnee 

2 


based on data (e.g., 
academic history 
and assessment 
scores) 

classes based on 
individual student 

survey results 

evaluation team 
annually 


on data 

Fern Creek 

2 


data 



1 = Moderate 

Freshmen or upperclassmen are 

Moore 

2 






assigned based on data 

Valley 

2 







2 = High 

Freshmen and upperclassmen 

Waggener 

2 







are assigned based on data 

Western 

2 


36 




Data 

Operational Data Source(s) Collection 

Key Elements of Definition for Measuring Schedule for 

Component for Indicator the Indicator the Indicator 

Raw Scoring and Scoring Criteria for Fidelit 

Possible 

Indicator 

Score Per Scoring School 

School Criteria Name 

y 

Indicator 

Score 




Academy @ 
Shawnee 

4 



3-6 = 

Fern Creek 

4 

Component Level Score 

0-6 

High Implementation 

Moore 

4 



0-2 = 

Valley 

3 



Low Implementation 

Waggener 

4 




Western 

4 

Implementation with fidelity = At least 67% of the schools have a minimum score of 3 
Implementation without fidelity = Fewer than 67% of the schools have a score of 3 or above 

100% of the schools have a 
minimum score of 3; thus, 
this component has been 
implemented with fidelity 
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Project Goal 2: Student Engagement in School and College Readiness 


The second i3 project goal is to provide a range of personalized supports to students to 
increase engagement in school and promote college readiness. The primary strategy is to create 
CAT advisory periods for students that are designed to focus on student engagement in school and 
college readiness. Additionally, each CAT facilitator, who is a teacher or other appropriate staff 
member working in the school, is charged with becoming an advocate for the small group of 
students in that advisory. To measure progress on this goal, a variety of indicators were identified. 
These indicators include the percentage of students who participate in a CAT advisory period, the 
percentage of students who meet ACT and PLAN benchmarks, postsecondary enrollment rates, and 
three scales from the CSS to assess students’ perceptions of adult social support, peer social support, 
and connectedness to their school 13 . CARTs were also asked to complete a survey focused on CAT 
implementation in the spring of 2015. Table 21 outlines the objectives for Project Goal 2 and the 
yearly or end-of-project targets, when appropriate. 


Table 21. Project Goal 2 Objectives, Performance Measures, and Targets 


Objective 

Performance Measure 

T arget 

A 

Provide a CAT advisory period for 
students 

Percentage of students who are in an 
advisory during the academic year 

100% 

B 

Increase students’ knowledge of and use of 
college-ready skills 

Percentage of students who reach 
ACT and PLAN benchmarks in 
English, reading, math, and science 

ACT by subject: 
English 38% 
Reading 30% 
Math 20% 
Science 19% 

PLAN by subject: 
English 50% 
Reading 32% 
Math 23% 
Science 19% 

Percentage of students enrolled in 
postsecondary institutions in the fall 
following high school graduation 

55% 

C 

Improve students’ sense of social support 
from adults 

CSCI Adult Social Support Scale 

3.42 

D 

Strengthen peer-to-peer relations and 
perceptions of peer social support 

CSCI Peer Social Support Scale 

3.52 

E 

Improve students’ perceptions of school 
connectedness and engagement 

CSCI School Connectedness/ 
Engagement Scale 

3.32 


This goal was particularly important to JCPS project staff because when the grant started, the 
state of Kentucky did not have a definition of “college and career readiness”. Shordy after the grant 


15 As describe previously, the 2015 CSS indicators replaced the CSCI indicators. Since the CSCI indicators are the recorded 
performance measures, however, they are still included on the objective table. 
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was funded, the Kentucky Department of Education transformed the system and called it “College 
and Career Readiness for All.” With this change, the statewide focus shifted towards career 
readiness, including career and technical education and certifications connected to career pathways. 

What the project director would like to concentrate on next is determining a way to measure 
“work readiness” factors, such as teaching students about punctuality and tenacity. Project staff 
said, “How do we find the balance between [academics and soft skills]? How do we help kids learn 
those skills [i.e., work readiness and soft skills]? And, how do we measure these skills?” Project 
staff continued by explaining the difficulties in accomplishing this, 

There are so many things that impact a student’s test scores. The bigger challenge is figuring out how 

to [do] that effectively in the research. You can ’t do one or the other, it has to be both. The 

challenge is to equip kids with skills they need. The outcomes are so convoluted with other factors. 

We have so much programming in place. Which lever do you push ? 

Objective A: College Access Time (CAT) 

CAT advisory periods, led by the CARTs assigned to each school, were fully implemented at 
all six i3 schools during the 2010-201 1 school year (project Year 1). Throughout the past five years, 
there have been several CART transitions at the schools. For Year 5, two of the six CARTs were 
new in their positions at the beginning of the year. In the spring of 2015, the six CARTs were asked 
to complete an online survey to provide insight as to how the CAT advisory periods were operating 
during the 2014-2015 school year. Of the six CARTs, four surveys were received for a response rate 
of 67%. Additionally, upon completing the survey, the CART s were invited to participate in a 
follow-up interview. Only one CART responded and completed an interview. The following 
sections highlight relevant information related to CAT implementation at the project schools during 
the 2014-2015 school year, summarized from information derived from both the spring survey and 
project staff interviews. 

Purpose of CAT 16 . Project staff commented on how the advisories started at the 
beginning of the project. The CART position was written into the grant with this position being 
responsible for developing the CAT advisory period curricula and training the teachers. To assist 
with implementation of CAT, Education Northwest was hired by JCPS to serve as the technical 
assistance provider. In partnership with JCPS, they put several deliverables together that evolved 
over time and project staff credit them with putting together meaningful professional development 
sessions for the CARTs. The first year of the i3 grant was spent training staff to build their capacity 
as leaders and gain their insight into what an “advisory” should be. Because the district is going to 
continue support of the CARTs at 19 schools throughout the district, project staff indicated that 
they believe they have justified the purpose of the CAT advisories. 


16 In past reports, information about CAT Design Teams was included. However, when asked if they have a Design Team, all 
four of the CARTs responded that the Design Teams are no longer being utilized so the related section has not been included 
in the Year 5 report. 
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Over the course of five years, district staff learned several lessons while implementing CAT 
in the project schools. First, project staff shared that they would develop a better on-boarding 
system to train new CARTs immediately as there was a lot of turnover with staff. Once the CARTs 
became trained, they were often promoted to other district positions, such as assistant principals. 
The CART position gave them experience with curriculum development, working across schools 
with teachers, as well as having a clear understanding of how to work with high school students. 
Second, the district would take a more active role in supporting the principal and CART so both 
understood the expectations and knew what resources were available. According to project staff, 
the success of the program depends on, in large part, how committed the principal is to the concept 
and the quality of the lesson that the CART produces. 

For sustainability purposes, project staff would also like to update the SharePoint site so it 
can continue to be used as an online lesson repository for all of the CARTs. Further, they would 
like to increase professional development and collaboration time. To accomplish this, JCPS district 
staff plan to utilize the Goal Clarity Coaches, which are in each school at least on a part-time basis. 
District staff envision the Goal Clarity Coaches as individuals who could organize these types of 
training. 

Efforts have already begun to document the process. JCPS recently created two documents 
for school leadership: the "‘College Access Time Instructional Framework for Schools leaders” and “Jefferson 
County Public Schools College Access Time Guidelines for School Leaders.” Additionally, Education 
Northwest created an “I Can Statements” document for JCPS. This document lists objectives, 
learning outcomes, and example “I Can” statements. The objectives included in the document are 
listed below: 

1 . Developing career interests and motivation 

2. Supporting application and financial assistance planning 

3. Developing college knowledge 

4. Developing 21st century skills 

5. Building teams and relationships 

6. Developing life skills 

7. Monitoring students’ academic progress 

8. ACT preparation 

9. Intervening with subject-specific remediation 

Table 22 shows the objective, learning outcomes, and example “I Can” statements for the 
Developing College Knowledge objective. 
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Table 22. Example “I Can” Statements for Developing College Knowledge 


Objective 

Learning Outcomes 

Example 1 Can Statements 

Developing 

Students will: 

1. 1 can explain basic needs for surviving 

college knowledge 

1. Learn how to “survive” college 

a. E.g. choosing a roommate, 
registering for classes, finding a 
tutor, and self-advocacy 

b. Time management 

in college that include things like 
choosing compatible roommates, 
registering for classes, finding support 
resources if 1 am having trouble, and 
asking for help when 1 need it. 


c. Differences between professor’s 
expectations vs. high school 
expectations (e.g., homework 

2. 1 can describe and demonstrate 

effective methods of managing time 
that works for me. 


practices and office hours) 

2. Develop cognitive skills important 
for success such as: 

3. 1 can develop effective arguments to 

support my opinions using facts and 
evidence. 


a. Discussion with adults and with 
peers 

4. 1 can apply things 1 know to solve 

problems and make arguments. 


b. Application of knowledge to 
new situations 

c. Note taking 

5. 1 can take notes to summarize main 

points from different types of 
presentations. 


On the spring implementation survey, each CART was asked to rank the four CAT priorities 
from most to least important. All four of the CARTs ranked “Strengthen peer-to-peer relations and 
perceptions of peer support, particularly within career themes” as the least important priority. Half 
of the CARTs ranked “Build knowledge and use of college-ready skills and habits” as the most 
important priority while one CART indicated “Improve students’ sense of affiliation with adults and 
peers” and another CART selected “Advance college -ready skills, monitor student progress, and 
increase students’ sense of affiliation with adults and peers” as the most important parts of CAT. 

The CART s were also asked to describe what they hoped to achieve during CAT by the end 
of the year. Responses follow: 

• During the past year, m hoped to instill in our younger students the importance of building a strong 
ivork ethic and maintaining a positive grade point average. As the students get closer to graduation, our 
goal is to have EVERY student have viable post high school opportunities, whether those opportunities 
are educational or vocational. 

• I want all students to believe that college is within reach. I want the older students to know what the 
steps are to achieving college acceptance and attendance. 

• Just to get through it. ft is a] waste of time, [neither the] teachers nor students like advisoy time. The 
classes that use it for end-of-course work get the most out of it. 

• Through advisory, I hoped to achieve a greater understanding of the roles and responsibilities of students 
in our building. Aiding them in finding purpose in their lives as individuals and students. Helpfng to] 
guide them into the next phases of their life. 

On the survey, CARTs were asked to reflect on how often CAT lessons follow the elements 
of the Classroom Instmctional Framework (CIF), an observation tool used throughout the district. 
Half of the CARTs responded that they almost ahvays use all four components of the CIF, which are: 
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(1) establishing engagement; (2) fostering connections; (3) deepening understanding; and (4) making 
meaning. On three of the components (i.e., establishing engagement, deepening understanding, and 
making meaning), one CART indicated that they seldom used any of those. 

Most of the CAT periods have curricula specific to each grade level. For example, freshmen 
and sophomores may spend more time on ACT or PLAN preparation, while juniors and seniors 
focus more on researching colleges, completing applications, and other activities related to 
postsecondary education and careers. Table 23 shows the curriculum descriptions for each grade 
level by the survey respondents. 


Table 23. CAT Descriptions by Grade Level 


9th Grade 

1 Oth Grade 

1 Ith Grade 

12th Grade 

75% of English teachers 
participate in CAT 
lessons/activities 

75% of English teachers 
participate in CAT 
lessons/activities 

1 00% of English teachers 
participate in CAT 
lessons/activities 

1 00% of English teachers 
participate in CAT 
lessons/activities 

Advisory time 

End-of-course 
preparation, special class 
preparation, or advisory 
time 

End-of-course 
preparation, special class 
preparation, or advisory 
time 

End-of-course 
preparation and special 
class preparation 

Ice breaker, school 
update, lesson (e.g., 
character development) 

Ice breaker, school 
update, lesson (e.g., 
advocacy and 
citizenship) 

Ice breaker, school 
update, lesson (e.g., 
preparing for college, 
grades, decisions, and 
applications) 

Ice breaker, school 
update, lesson (e.g., 
preparing for the end of 
high school, college 
choice applications, etc.) 

The ninth grade 
emphasizes maintaining 
high grades, study skills, 
and proper school 
etiquette. Students may 
be selected to attend a 
campus visit. 

The 1 Oth grade 
continues to emphasize 
good grades but also 
adds the importance of 
ACT test scores, AP 
classes, and 
extracurriculars. 
Students are afforded 
the opportunity to 
attend multiple college 
visits. 

The 1 1 th grade 
continues to emphasize 
all of the lessons from 
ninth and 1 Oth grade 
with an added emphasis 
on: college selection 
(including college visits, 
both on and off campus), 
introduction to financial 
aid, and ACT prep. 

The 1 2th grade 
emphasizes all items 
from ninth, 10th, and 
1 Ith grade with an 
emphasis on filling out 
college applications, 
filling out the Free 
Application for Federal 
Student Aid (FAFSA), 
and economic 
competence (e.g., real- 
world personal finance). 


CAT Monitoring. The CARTs were also asked if they use the walkthrough instrument 
that was developed for CAT. Two of the CARTs replied that they do not use it, while one CART 
indicated that it was used two or three times a month and another CART said the school used it 
monthly. One CART added that advisory times were monitored through teacher feedback from the 
lessons. 


Further, the CARTS were asked how often students had the opportunity to provide formal 
feedback about CAT. Again, 50% of the CARTs who completed the survey responded that the 
students are never given an opportunity to provide formal feedback while another CART said that 
they have an opportunity to do so once a year. The other CART who has collected student feedback 
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commented that “Students are constantly encouraged to provide feedback about mentors and 
mentoring sessions.” 

CAT Advisors. Two of the four CARTs said that they trained advisors during the 
2014-2015 school year for an hour, with one adding that follow-up was provided as needed. For the 
CARTs who did not provide training, their reasons were that only an eighth of the staff serve as 
advisors and the CART only started the position in September. The other CART said that it was 
not applicable as “most [of the advisors] were teaching content during advisory.” 

The CARTs were also asked to explain factors that facilitated advisor effectiveness as well as 
challenges faced by the advisors. Table 24 shows the results of those questions. Facilitating factors 
include the concept of CAT, positive relationships with students, and supportive administration. In 
contrast, challenges include apathetic students, time, and staff buy-in. 


Table 24. Facilitating and Challenging Factors Faced by Advisors 


Success Factors 

Challenges 

College-going knowledge 

Conducting advisory in an actual English class 

Passion for kids 

No grade so kids don’t care 

Positive relationships with the students 

Students with negative preconceived ideas of their 
future 

Using advisory for end-of-course exams 

Time 

A willingness to follow through with their students 

Buy-in 

Consistent implementation 

Lack of buy-in. It is someone else’s job 

Rapport-building 

Time to effectively mentor their students 

A supportive administration 



CAT Minutes and Ratio by School. The original requirements of the grant were for 
CAT advisory periods to be held 55 minutes per week with a student-to-staff ratio of 20:1. Table 25 
shows the frequency and duration of each CAT period as reported on the spring 2015 survey. Based 
on these data, two schools clearly met the goal of having the CAT advisories last at least 55 minutes 
and meeting approximately weekly. All of the schools met the intended ratio of one advisor to 20 or 
fewer students. 


Table 25. CAT Logistics by School 


Frequency, 
Duration, and Ratio 

School#! 

School #2 

School #3 

School #4 

Frequency of meetings 

Two or three 
times a month 

Weekly 

Weekly 

Students meet in 
smaller mentoring 
groups periodically. 
It differs from 
student to student. 
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Frequency, 
Duration, and Ratio 

School#! 

School #2 

School #3 

School #4 





We do not have a 

Number of weeks 

Fewer than 1 5 

30 to 34 weeks 

More than 34 

specific CAT time. 
Instead, we have 

during school year 

weeks 

weeks 

more individualized 





mentoring sessions 
with our students. 

Minutes 

Depends on the 
lesson/activity 

55 minutes 

60 minutes 

See above 


1:30 

(it is a regular class) 



Less than 1:10; 

Student-to-staff ratio 

1:18 

1:20 

approximately 1:6 
or 1:8 


CAT Training and Coaching Sessions. Only two of the CARTs participated in JCPS 
training sessions during the 2014-2015 school year. Those two CARTs participated in one training, 
which included topics on: (1) What is CAT; (2) What we are doing; and (3) Where to find resources. 
The CARTs listed the following as topics they would like to see as future training topics: 

1 . Building a college-going culture 

2. Generating buy-in 

3. ACT preparation 

4. College-focused lessons 

5. Financial aid workshops 

6. College selection process 

Although only two CARTs attended the training, three of the CARTs disagreed that the 
training sessions were valuable. One CART agreed that the SharePoint resource provided by JCPS 
was helpful while two CARTs disagreed and one CART strongly disagreed. Finally, three out of the 
four CARTs disagreed that the coaching sessions were valuable while one CART agreed that the 
coaching sessions were helpful. 

Factors Facilitating and Challenging Implementation. CARTs were also asked to list 
two or three facilitating and challenging factors regarding CAT implementation at their schools. 
Table 26 lists those factors. Although some of the CARTs listed teacher buy-in and engagement as 
an ongoing challenge, a few of the CART s thought that they had seen an increase in teachers’ 
acceptance of the CAT program. Many of the successes appear to result from changes that the 
CARTs have made to the program, such as lesson design and re -branding CAT under a new name. 
Challenges include external factors, such as time and scheduling issues. Other challenges that 
continue to plague the project schools include buy-in from both students and teachers. The 
comments also indicate that it is difficult to train new staff, both teachers and administrators, in 
order to fully engage and understand the CAT concept. 
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Table 26. Facilitating and Challenging Factors of CAT Implementation 


Success Factors 

Challenges 

Our junior English teachers are super efficacious 
with college access activities and materials. 

All teachers/mentors are not created equal. Some 
teachers/mentors are wonderful with their students 
(e.g., time, follow-up, support, etc.) while others 
(no matter how much support you provide the 
MENTOR) merely are satisfied with doing the bare 
minimum. 

Student buy-in 

Buy-in. Advisory was not done well in this building 
prior to my arrival so teachers and administrators 
have a bad taste in their mouths. 

Using for end-of-course content 

Getting teacher buy-in 

We moved to a much smaller, more individualized 
approach to student mentoring this past school year. 

Buy-in 

Journalism use of video information 

Apathy 

Our senior English teachers are great at building 
rapport and marketing the importance of college. 

Many teachers in the building do not teach the 
“whole” child. They teach content over students. It 
makes it hard to sell college and not be able to build 
rapport. 

We have a very supportive administration when it 
comes to implementing new ideas and new ways of 
building a true “col lege -going” environment. 

Teachers not doing lessons 

The second biggest challenge we face is overcoming 
many of the outside influences on our students. 

Many people in the students’ “outside school” life do 
not support our ideas of what is needed for success 
and work to undermine the students’ future. 

Being stretched in multiple directions 


Student Perceptions of CAT. Students were asked several items related to CAT on the 
2015 Student Survey to gain an understanding about what students perceived they were learning 
during CAT and the relationships they have developed with their advisor and peers within their 
CATs. Table 27 shows student responses by grade level regarding what they have learned in the 
CAT advisory periods. As expected, due to the focus of the CAT advisories at each grade level, the 
means are greater for upperclassmen on the items related to college preparation. For example, the 
mean for the item, “I have learned how to complete a college application” is 2.53 for seniors, 
compared to 1.51 for freshmen. 

Table 27. Student CAT Perceptions by Grade Level 


9th Grade 10th Grade Nth Grade 12th Grade 
N M SD N M SD N M SD N M SD 


CAT has helped me . . . 


understand the college admission 
process. 

680 

1.97 

1.46 

554 

2.03 

1.36 

564 

2.30 

1.24 

413 

2.41 

1.22 

prepare for the ACT and/or PLAN 
test. 

683 

1.99 

1.43 

552 

2.25 

1.30 

565 

2.62 

1.17 

409 

2.43 

1.23 
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9th Grade 

1 Oth Grade 

1 Ith Grade 

1 2th Grade 


N 

M 

SD 

N 

M 

SD 

N 

M 

SD 

N 

M 

SD 

develop life skills (e.g., time 
management, communication, conflict 
resolution, and financial literacy). 

678 

2.06 

1.40 

549 

2.22 

1.31 

565 

2.40 

1.18 

405 

2.32 

1.23 


During my past and current experiences in CAT . . . 


1 have had opportunities to research 
and develop career interests. 

680 

2.18 

1.38 

550 

2.27 

1.33 

567 

2.50 

1.15 

410 

2.51 

1.17 

1 have learned how to complete a 
college application. 

681 

1.51 

1.33 

548 

1.68 

1.32 

567 

1.93 

1.28 

408 

2.53 

1.24 

1 have learned about financial literacy, 
financial aid, and paying for college. 

680 

1.75 

1.40 

550 

1.85 

1.33 

565 

207 

1.26 

410 

2.52 

1.20 

1 have created a resume. 

676 

1.40 

1.35 

552 

1.69 

1.39 

566 

1.91 

1.32 

41 1 

2.17 

1.31 

1 have learned how to apply for a job. 

674 

1.65 

1.48 

549 

2.14 

1.48 

563 

2.26 

1.33 

408 

2.45 

1.34 


Principal Perceptions of CAT. Principals were asked a series of questions regarding the 
CAT advisory periods on the spring 2015 survey. Of the three principals who responded, they had 
specific goals that they wanted to accomplish within the CAT program during the 2014-2015 school 
year: 


• [ The CAT should result in having j one adult in the building that students feel connected to and small 
groups of same sex/ same age peers that students have built a relationship with. [W ? need] college 
awareness and explanations of personal accountability with college and career readiness majors for 
freshmen. Senior interview scores of a four. One hundred percent of the [. students to submit] applications 
to college. As close to 100% as possible [of students having! FAFSAs completed. 

• Students will understand what it means to be college and career ready. Students will also be exposed to a 
variety of college readiness lessons and be prepared to apply to colleges and to understand terminology used 
in colleges. Students will know [that] they will be able to go to college, even if their families cannot afford 
to send them to college. 

• We use academic advising tune every six weeks. We use the time for teachers to communicate learning 
goals with each student. We, in addition, use our college-going culture initiative in our English classes. 

Principals were also asked to list factors that facilitated CAT implementation as well as any 
challenges that the school faced with implementing the advisory periods. As shown in Table 28, the 
principals listed a variety of success factors, including building relationships, curriculum, and 
staffing. Similar challenges to that of the CART s were listed, which included teachers not buying 
into the lessons and the engagement of all advisors. 
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Table 28. Principal Reports of Facilitating and Challenging Factors of CAT Implementation 


Success Factors 

Challenges 

Authentic relationships 

Building relationships 

Effective differentiated lesson-planning 

Making sure all lessons have relevance to the 
students 

Pairing students with the right teacher 

Teachers who do not buy into lessons 

Implemented curriculum 

Ensuring all advisors are engaged and following 
through with the plan 

Meaningful lessons 

Lessons that reach all learners 

Organization by someone not in the classroom 
full-time 


Goal-setting towards college 

Time 

Implementation with fidelity 



Objective B. Increase Students ’ Knowledge of and Use of College-Ready Skills 

For the purpose of the evaluation, end-of-project ACT and PLAN targets were established 
based on 2009-2010 data. At baseline (2009-2010), there was approximately a 15 percentage point 
gap between the rates of students meeting benchmarks in i3 schools relative to the district mean in 
all subjects except science. For science, the gap was about 10 percentage points. Since the i3 
project’s aim was to close these gaps, the end-of-project targets were defined as the 2009-2010 
content area ACT or PLAN means across the i3 schools plus 15 percentage points. Table 29 shows 
the established end-of-project targets for student performance on the ACT and PLAN exams. 


Table 29. ACT and PLAN End-of-Project Targets 


Content Domains 

ACT 

End-of-Project Target 

PLAN 

End-of-Project Target 

English 

38% 

50% 

Mathematics 

20% 

23% 

Reading 

30% 

32% 

Science 

19% 

19% 


During the 2014-2015 school year, a total of 1,404 lOth-grade and 1,318 1 lth-grade students 
from the i3 project schools took the PLAN and ACT, respectively. In this report, evaluators 
examined the extent to which students in the i3 schools met ACT and PLAN benchmarks. State 
and district-level data and findings from the Years 1 through 4 (2010-2011, 2011-2012, 2012-2013 
and 2013-2014) are included to show the i3 schools’ progress trajectories over time. In addition, 
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school differences in relation to the ACT and PLAN outcomes are investigated using chi-square 
tests. 1, Cramer’s V 18 is reported as an indicator of effect size. 

Subgroup analyses (e.g., analyses based on students’ demographic characteristics) were 
conducted to investigate whether the percentages of students meeting or exceeding the ACT and 
PLAN benchmarks differed by student attributes. Since students were nested within schools (i.e., 
within a multilevel structure), and the percentages of students meeting or exceeding the benchmarks 
seemed to differ by school (Nixon et al., 2012), a series of hierarchical generalized linear models 111 
and subsequent analyses were conducted. The goal was to examine the associations between student 
attributes (i.e., gender, race, FRPM status, and LEP status) and the percentages of students passing 
the benchmarks for each subject area. 

PLAN as an Indicator. As mentioned, PLAN scores serve as an early indicator of 
students’ college readiness (relative to ACT scores). Figure 9 shows the percentages of students 
meeting or exceeding the PLAN benchmarks in English, math, reading, and science across the state, 
district, and i3 schools in project Year 5 (2014-2015). Overall, the percentages of students meeting 
or exceeding the benchmarks were lower in STEM-related subjects (i.e., math and science), which is 
consistent with district and state-level data from the 2013-2014 school year 20 . In addition, the 
percentages of i3 students meeting or exceeding the PLAN benchmarks were lower than findings 
from state and district-level data. 


17 Base on chi-square cell distributions, pairwise chi-squares were conducted to determine school differences in the percentages 
of students meeting or exceeding the benchmarks. 

18 To interpret Cramer’s V, Ferguson (2009) suggests that a value of 0.20 was the recommended minimum effect size 
representing a “practically” statistically significant effect for social science data; a value of 0.50 presents a moderate effect size; 
and a value of 0.80 presents a strong effect size. 

19 When between-school variations in student outcomes are evident, multilevel modeling is a preferred approach because it 
adjusts for these variations. 

20 State and district data for 2014-2015 are not yet available. Therefore, the 201 3-2014 performance data is used for 
comparison. 
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100 % 


75% 
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English Math Reading Science 

B State Y5 B District Y5 B i3 Schools Y5 


Figure 9. Percentages of students from i3 schools meeting or exceeding PLAN benchmarks 
across the state, district, and i3 schools in project Year 4. 21 

Compared to Year 4 findings, i3 schools’ performance declined in all subject areas in Year 5 
(see Figure 10). 22 Although the schools still have not reached the project targets, the gaps were 
reduced across all subject areas. Gaps between performance and end-of project goals ranged from 
13 to 19 percentage points. The gap in English increased to 19% in Year 5, whereas the Year 4 
results brought project schools within two percentage points of the end-of-project goal. The 
percentage of individuals to achieve the English benchmark in Year 5 (2014-2015) is at an all-time 
low since the baseline measurement (2009-2010). The gap in science scores resulted in the lowest 
difference from Year 4 to Year 5, which is also the smallest gap from the end-of-project goal. 


21 Source for the state and district data: http://education.ky.gov/AA/Reports/Pages/EXPLORE-and-PLAN-Data.aspx 

22 There are some discrepancies in the percentages of students passing the benchmarks between the reports archived at the 
Kentucky Department of Education and data provided byJCPS. The discrepancies may be due to various sample sizes reported 
in different data sources, in this report, the data provided byJCPS are used. 
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Baseline (2009-20 1 0) ■ Year I (20 1 0-20 II) ■ Year 2 (20 1 I -20 1 2) ■ Year 3 (20 1 2-20 1 3 

■ Year 4 (20 1 3-20 14) ■ Year 5 (2014-20 1 5) ■ End-of-project target ■ Gap to Target 

Figure 10. Percentages of students from i3 schools meeting PLAN benchmarks overtime. 

Overall, the percentage of students meeting or exceeding the benchmarks differed by school 
across all subject areas. Students in the i3 schools made the most progress in the STEM subjects 
(i.e., math and science) on the PLAN assessment. All project schools experienced decreases in the 
percentage of students achieving the benchmarks for English and reading. None of the schools met 
the end-of-project goals in any subject area. Table 30 shows the changes in the percentage of 
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students meeting or exceeding the benchmarks by school over time. The data shown in bold font 
with a gray background suggest positive outcomes, such as progress made by schools over time as 
well as progression toward the end-of-project targets. Findings are summarized as follows: 

• PLAN English: All six i3 project schools showed decreases in percentage points from 
Year 4 to Year 5. Fern Creek Traditional Fligh School exceeded the end-of-project 
target in Year 4, but dropped below the goal in Year 5 with a decrease of 13 percentage 
points. The gaps to the end-of-project target range from 7% to 37%. Waggener Fligh 
School retained the least amount of variance from Year 4 to Year 5, though the school 
remains 20 percentage points under the end-of-project target. 

• PLAN Math: Three of the six i3 schools (i.e., Moore Traditional Fligh School, Valley 
Fligh School, and Waggener Fligh School) increased their math percentage points 
between Year 4 and Year 5. The remaining three schools (i.e., Academy @ Shawnee, 
Fern Creek Traditional Fligh School, and Western Fligh School) had slight decreases, 
ranging from two to five percentage points. All of the i3 schools were still more than 
nine percentage points from the end-of-project target, with gaps between 10 and 

21 percentage points below the target. 

• PLAN Reading: Similar to the findings of PLAN English, all of the i3 schools showed 
a decline in PLAN reading scores between Years 4 and 5. Valley Fligh School 
experienced the largest decrease in benchmark achievement with a decrease of 12 
percentage points. Additionally, none of the schools met or exceeded the end-of-project 
target; the gaps from the target range from 10 to 19 percentage points below the goal. 

• PLAN Science: All six i3 schools, except Western Fligh School, showed an increase in 
meeting or exceeding the science benchmark from Year 4 to Year 5. Western Fligh 
School did not show improvement, but the decline from Year 4 to Year 5 was minimal 
(0.5%). Although the majority of schools improved in Year 5, none of the i3 schools 
met the end-of-project targets. The percentage of students failing to meet the 
benchmark in science ranges from 4 to 14 percentage points. 

Overall, the majority of i3 students did not meet the end-of-project targets. The 
benchmarks for PLAN reading and PLAN science were adjusted prior to the 2014-2015 assessment 
(i.e., Year 5). The PLAN reading benchmark was increased from 17 to 18, while the PLAN science 
benchmark decreased from 21 to 20. Project and school staff should continue to investigate the 
reasoning behind the gaps and develop/identify strategies to prevent further decline. 
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Table 30. Percentages of Students Meeting or Exceeding the PLAN Benchmarks by School 
Over Time 


Project Year 

Academy 

@ 

Shawnee 

Fern 

Creek 

Moore 

Valley 

Waggener 

Western 

* 2 ( 5 ) 

Effect 

Size 

English (End-of-Project Target = 50%) 

Year 1 

22.0 

48.9 

33.0 

31.6 

34.2 

21.5 

63.63*** 

- 

Year 2 

25.3 

55.8 

51.6 

44.7 

41.8 

25.0 

55.33*** 

0.228 

Year 3 

38.3 

51.5 

42.2 

35.6 

34.2 

25.7 

4 1 .04*** 

0.170 

Year 4 

43.3 

56.1 

42.3 

33.4 

31.9 

28.7 

65.56*** 

0.205 

Year 5 

28.6 

43.0 

35.6 

24.9 

30.4 

13.4 

57.54 

0.202 

% Change from 
Yl to Y2 

3.3 

6.9 

18.6 

13.1 

7.6 

3.5 


% Change from 
Y2 to Y3 

13.0 

-4.3 

-9.4 

-9.1 

-7.6 

0.7 


% Change from 
Y3 to Y4 

5.0 

4.6 

0.1 

-2.2 

-2.3 

3.0 


% Change from 
Y4 to Y5 

-14.7 

-13.1 

-6.7 

-8.5 

-1.5 

-5.3 



Gap to the End-of- 
Project Target 

-21.4 

-7.0 

-14.4 

-25.1 

-19.6 

-36.6 



Math (End-of-Project Targe 

t = 23%) 







Year 1 

2.3 

12.8 

8.0 

5.3 

5.5 

3.2 

31.21*** 

- 

Year 2 

13.3 

30.3 

25.5 

8.5 

26.1 

10.3 

5 1 .78*** 

0.221 

Year 3 

4.3 

1 1.3 

6.7 

5.8 

4.4 

2.9 

20.46** 

0.120 

Year 4 

7.5 

15.5 

8.5 

5.5 

5.6 

7.0 

29.68*** 

0.138 

Year 5 

2.3 

13.5 

10.3 

7.9 

9.8 

3.5 

1 5.07** 

0.104 

% Change from 
Yl to Y2 

1 1.0 

17.5 

17.5 

3.2 

20.6 

7.1 


% Change from 
Y2 to Y3 

-9.0 

-19.0 

-18.8 

-2.7 

-21.7 

-7.4 



% Change from 
Y3 to Y4 

3.2 

4.2 

1.8 

-0.3 

1.2 

0.1 


% Change from 
Y4 to Y5 

-5.2 

-2.0 

1.8 

2.4 

4.2 

-3.5 

Gap to the End-of- 
Project Target 

-20.7 

-9.5 

-12.7 

-15.1 

-13.2 

-19.5 



Reading (End-of-Project Target = 32%) 

Year 1 

8.3 

32.6 

26.2 

20.6 

21.6 

13.2 

51.35*** 

- 

Year 2 

18.1 

48.3 

41.6 

35.8 

37.0 

22.6 

44 12*** 

0.204 

Year 3 

12.1 

27.0 

20.4 

15.3 

16.2 

9.7 

34.3 1 

0.155 

Year 4 

17.5 

31.8 

30.3 

20.7 

16.0 

14.8 

42 42 *** 

0.165 

Year 5 

14.3 

22.0 

19.5 

9.0 

15.8 

13.4 

23.20*** 

0.129 
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Project Year 

Academy 

@ 

Shawnee 

Fern 

Creek 

Moore 

Valley 

Waggener 

Western 

X 2 (5) 

Effect 

Size 

% Change from 
Yl to Y2 

9.8 

15.7 

15.4 

15.2 

15.4 

9.4 


% Change from 
Y2 to Y3 

-6.0 

-21.3 

-21.2 

-20.5 

-20.8 

-12.9 



% Change from 
Y3 to Y4 

5.4 

4.8 

9.9 

5.4 

-0.2 

5.1 


% Change from 
Y4 to Y5 

-3.2 

-9.8 

-10.8 

-1 1.7 

-0.2 

-1.4 



Gap to the End-of- 
Project Target 

-17.7 

- 10.0 

-12.5 

-23.0 

-16.2 

-18.6 



Science (End-of-Project Target = 19%) 

Year 1 

2.3 

6.9 

4.4 

2.6 

5.0 

1.4 

14.57* 

- 

Year 2 

7.2 

24.0 

23.0 

1 1.8 

21.2 

12.3 

25.73*** 

0.156 

Year 3 

1.4 

9.7 

6.2 

4.7 

3.9 

3.4 

19.45* 

0.1 17 

Year 4 

6.7 

9.5 

6.7 

5.2 

3.8 

5.2 

9.87 

0.080 

Year 5 

7.5 

15.4 

1 1.9 

7.2 

8.2 

4.7 

22.38*** 

0.126 

% Change from 
Yl to Y2 

4.9 

17.1 

18.6 

9.2 

16.2 

10.9 


% Change from 
Y2 to Y3 

-5.8 

-14.3 

-16.8 

-7.1 

-17.3 

-8.9 



% Change from 
Y3 to Y4 

5.3 

-0.2 

0.5 

0.5 

- 0.1 

1.8 


% Change from 
Y4 to Y5 

0.8 

5.9 

5.2 

2.0 

4.4 

-0.5 

Gap to the End-of- 
Project Target 

-1 1.5 

-3.6 

-7.1 

-1 1.8 

-10.8 

-14.3 




Note. Data for Year I were obtained from the Year I Report (Nixon et al., 20 1 2); Data for Year 2 were obtained from the 
Year 2 Report (Donahue et al., 20 1 2); and Data for Year 3 were obtained from the Year 3 Report (Donahue et al., 20 1 3). 
Effect size was calculated by using Cramer’s V (Ferguson, 2009): a value of 0.20 was the recommended minimum effect size 
representing a "practically” significant effect for social science data; a value of 0.50 presents a moderate effect size; and a 
value of 0.80 presents a strong effect size. 

*** p < 0.001 
** p < 0.0 1 

*p< 0.05 


Statistically significant school variations in the percentage of students meeting PLAN 
benchmarks suggested that it is more appropriate to use multilevel modeling to examine the 
associations between student attributes and the percentages of students meeting or exceeding the 
benchmarks. Table 31 shows the results of the hierarchical generalized linear models. 
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Table 31. Results of Hierarchical Generalized Linear Models: PLAN Outcomes® 


Fixed Effect 

English 
p OR 

Math 

p OR 

Reading 
p OR 

Science 
p OR 

Female a 

0.31* 

1.36 

-0.29 

0.75 

0.20 

1.22 

-0.43* 

0.65 

Students with free meal 
status b 

-0.41* 

0.66 

-0.67** 

0.51 

-0.37 

0.69 

-0.32 

0.73 

Students with reduced- 
price meal status c 

0.07 

1.07 

-0.49 

0.61 

0.29 

1.34 

0.97 

1.08 

Hispanic d 

-0.46 

0.63 

-0.37 

0.69 

0.07 

1.08 

-0.33 

0.72 

Black e 

-I JO*** 

0.33 

- 1 .66*** 

0.19 

- 1 .07*** 

0.34 

- 1 .46*** 

0.23 

Other Race f 

-0.09 

0.92 

0.26 

1.30 

-0.16 

0.85 

0.26 

1.29 


a Male is the reference group. 

b&c Students with paid meal status is the reference group, 
d, e, & f White students is the reference group. 


P= coefficient; OR = odds ratio 

The interpretation of odds ratio is that the odds of passing the benchmark is the degree to which the score deviates from I if 
the student was a “predictor = I (i.e., female, students with free meal status, students with reduced-price meal status, 
Hispanic, Black, and Other Race) in comparison to “predictor = 0 (i.e., male, students with paid meal status, and White).” If 
the odds ratio is greater than I, the predictor is more likely than the comparison to meet or exceed benchmarks; if the odds 
ratio is below I, the predictor is less likely than the comparison to meet or exceed benchmarks. 

***p < 0.00 1 
**p < 0.0 1 

*p < 0.05 


As shown in Table 31, the results suggest that students’ college readiness skills as measured 
by the PLAN differed by student characteristics across different subject areas. Specifically, Black 
students have lower odds of meeting or exceeding the benchmarks across all subject areas than 
White students; female students have higher odds of meeting or exceeding the benchmarks in 
English while they have lower odds of meeting or exceeding the benchmarks in science as compared 
to male students. Students with free meal status had lower odds of meeting or exceeding the 
benchmarks in English and math as compared to students with paid meal status. Findings for each 
subject area follow: 

• The odds of meeting or exceeding the English benchmark was 1.36 times higher if the 
student was a female as opposed to a male. In contrast, the odds of meeting or 
exceeding the science benchmark was 0.65 times lower if the student was a female as 
opposed to a male. 

• The odds of meeting or exceeding the English and math benchmarks was 0.66 and 
0.51 times lower for students qualifying for free meal status as opposed to students with 
paid meal status. 

• The odds of meeting or exceeding the English, math, reading, and science benchmarks 
was 0.33, 0.19, 0.34, 0.23 times lower for a Black student as opposed to a White student. 

ACT as an Indicator. Results of ACT scores are utilized as indicators of students’ college 
readiness for 11th graders. Figure 1 1 shows the percentages of students meeting the ACT 
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benchmarks in English, math, reading, and science across the state, district, and i3 schools during 
the fifth year of project implementation. The ACT benchmarks for mathematics and reading were 
adjusted from 22 to 19 and 20, respectively. Additionally, ACT introduced a science benchmark of 
23 for demonstrating college readiness in biology. The percentage of students meeting the 
benchmarks was lowest for science and math across the state, district, and project schools. 

100 % 

75% 


55.9% 



English Math Reading Science 

■ State Y5 ■ District Y5 ■ i3 Schools Y5 


Figure I I. Percentages of students meeting or exceeding ACT benchmarks 
across the state, district, and i3 schools in project Year 5. 

Note: State data were collected from the Kentucky State Report Card 

(http://applications.education.ky.gov/SRC/AssessmentByState.aspx) and the district data were collected from the Jefferson 
County Report Card (http://applications.education.ky.gov/SRC/AssessmentByDistrict.aspx). 

Relative to the established end-of-project targets. Figure 12 shows that the i3 schools fell 
short of all four subject area targets 21 . The percentage of i3 students meeting or exceeding the ACT 
benchmarks decreased in English and reading from Year 4 to Year 5 but increased in math. Overall, 
the gap to the end-of-project target ranges from 6% to 14%, with the largest gap being in reading, 
followed by science, math, and English. 


23 Science scores were not available for the Year 4. 
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Baseline (2009-20 1 0) ■ Year I (20 1 0-20 II) ■ Year 2 (20 1 I -20 1 2) ■ Year 3 (20 1 2-20 1 3 

B Year 4 (20 1 3-20 14) ■ Year 5 (2014-20 1 5) ■ End-of-project target ■ Gap to Target 

Figure 1 2. Percentages of students from i3 schools meeting 
ACT benchmarks across years. 

Overall, the percentage of students meeting or exceeding the benchmarks differed by school 
across all subject areas. Table 32 shows changes in the percentage of students meeting or exceeding 
the ACT benchmarks by school over time. The data shown in bold font with a gray background 
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suggest positive outcomes, such as progress made by schools over time as well as progression 
toward the end-of-project targets. Findings are summarized in the following bullets. 

• ACT English: Four of the six i3 schools (Academy @ Shawnee, Fern Creek Traditional 
Fligh School, Moore Traditional High School, and Western High School) showed 
increases from Year 4 to Year 5, with the largest increase of five percentage points being 
attained by Academy @ Shawnee. The other two i3 project schools (Valley Traditional 
High School and Waggener Traditional High School) showed decreases ranging from 
four to eight percentage points between Year 4 and Year 5. Fern Creek Traditional High 
School was the only school to exceed the end-of-project target. 

• ACT Math: All of the project schools failed to meet the end-of-project target for the 
percentage of students meeting or exceeding the math benchmark. Four of the six i3 
schools (Academy @ Shawnee, Fern Creek Traditional High School, Valley High School, 
and Western High School) increased the percentage of students exceeding the 
benchmark from Year 4 to Year 5. Valley High School and Western Traditional High 
School failed to meet the benchmark by approximately 1 3 percentage points each. 

• ACT Reading: Two project schools (Academy @ Shawnee and Moore Traditional High 
School) showed increases in the percentage of students meeting or exceeding the reading 
benchmark from Year 4 to Year 5; Academy @ Shawnee showed the greatest 
improvement with a 5.4% increase. However, all of the i3 project schools failed to meet 
the end-of-project targets by 3% to 22%. 

• ACT Science: While all of the schools failed to meet the end-of-project targets by 3% to 
18%, they did demonstrate increases in meeting the science benchmark from Year 4 to 
Year 5. The increases in percentage points range from one to seven, with the 
Academy @ Shawnee showing the most improvement. 

Fern Creek Traditional High School seemed to perform better than the other schools across 
all ACT subjects, while Academy @ Shawnee showed the most improvement across all subject areas 
from Year 4 to Year 5. In Year 5, Western High School continued to stmggle in STEM-related 
subject areas (i.e., math and science). Overall, the majority of students at the i3 project schools were 
struggling to meet the end-of-project targets across all subject areas. 


Table 32. Percentages of Students Meeting or Exceeding the ACT Benchmarks by School 
Over Time 


Project Year 

Academy 

@ 

Shawnee 

Fern 

Creek 

Moore 

Valley 

Waggener 

Western 

X 2 (5) 

Effect 

Size 

English (End-of-Project Target = 38%) 

Year 1 

9.6 

34.3 

31.7 

22.6 

27.2 

1 1.8 

43.60 * 

-- 

Year 2 

9.6 

34.7 

32.3 

22.3 

27.2 

12.2 

43.69*** 

0.200 

Year 3 

21.6 

49.9 

30.0 

20.0 

25.9 

25.0 

72.6 1 *** 

0.261 
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Project Year 

Academy 

@ 

Shawnee 

Fern 

Creek 

Moore 

Valley 

Waggener 

Western 

X 2 (5) 

Effect 

Size 

Year 4 

24.4 

45.6 

30.2 

28.2 

34.8 

19.8 

43.7 1 *** 

0.194 

Year 5 

28.9 

46.3 

30.4 

24.7 

27.3 

23.7 

48.22*** 

0.191 

% Change from 
Yl to Y2 

0.0 

0.4 

0.6 

-0.3 

0.0 

0.4 


% Change from 
Y2 to Y3 

12.0 

15.2 

-2.3 

-2.3 

-1.3 

12.8 


% Change from 
Y3 to Y4 

2.8 

-4.3 

0.2 

8.2 

8.9 

-5.2 


% Change from 
Y4 to Y5 

4.5 

0.7 

0.2 

-3.5 

-7.5 

3.9 


Gap to the End-of- 
Project Target 

-9.1 

8.3 

-7.6 

-9.8 

-10.7 

-14.3 



Math (End-of-Project Targe 

t = 20%) 







Year 1 

8.4 

16.8 

1 1.2 

2.2 

13.0 

4.3 

35.34**’ 

- 

Year 2 

8.4 

17.0 

1 1.8 

2.1 

13.6 

4.5 

35.66*** 

0.180 

Year 3 

1 1.4 

18.5 

12.5 

5.7 

13.8 

4.5 

30.21*** 

0.168 

Year 4 

5.2 

16.7 

14.5 

5.9 

12.0 

5.9 

27.04*** 

0.153 

Year 5 

12.2 

18.5 

13.5 

7.5 

10.8 

7.5 

22.83*** 

0.132 

% Change from 
Yl to Y2 

0.0 

0.2 

0.6 

- 0.1 

0.6 

0.2 


% Change from 
Y2 to Y3 

3.0 

1.5 

0.7 

3.6 

0.2 

0.0 

% Change from 
Y3 to Y4 

-6.2 

-1.8 

2.0 

0.2 

-1.8 

1.4 


% Change from 
Y4 to Y5 

7.0 

1.8 

1.0 

1.6 

-1.2 

1.6 


Gap to the End-of- 
Project Target 

-7.8 

-1.5 

-6.5 

-12.5 

-9.2 

-12.5 



Reading (End-of-Project Target = 30%) 

Year 1 

3.6 

24.2 

19.3 

14.0 

19.0 

7.5 

34.72*** 

- 

Year 2 

3.6 

24.3 

19.3 

13.9 

19.6 

7.7 

33.95*** 

0.180 

Year 3 

14.8 

27.2 

8.8 

15.7 

19.8 

12.2 

33.36’** 

0.177 

Year 4 

1 1.3 

26.6 

19.2 

15.5 

20.9 

13.2 

MM A F» 

22.45 

0.139 

Year 5 

16.7 

26.7 

14.3 

1 1.4 

14.9 

8.1 

42.36*** 

0.179 

% Change from 
Yl to Y2 

0.0 

0.1 

0.0 

- 0.1 

0.6 

0.2 


% Change from 
Y2 to Y3 

1 1.2 

2.9 

-10.5 

1.8 

0.2 

4.5 


% Change from 
Y3 to Y4 

-3.5 

-0.6 

10.4 

-0.2 

0.9 

1.0 
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Project Year 

Academy 

@ 

Shawnee 

Fern 

Creek 

Moore 

Valley 

Waggener 

Western 

X 2 (5) 

Effect 

Size 

% Change from 
Y4 to Y5 

5.4 

0.1 

-4.9 

-4.1 

-6.0 

-5.1 



Gap to the End-of- 
Project Target 

-13.3 

-3.3 

-15.7 

-18.6 

-15.1 

-21.9 



Science (End-of-Project Target = 19%) 

Year 1 

2.4 

8.4 

6.8 

1.6 

7.1 

1.9 

17.80 

- 

Year 2 

2.4 

8.6 

6.8 

1.6 

7.6 

1.9 

1 8.28 

0.130 

Year 3 

6.8 

8.4 

3.1 

3.3 

7.8 

2.6 

12.83* 

0.1 10 

Year 4 

4.4 

10.2 

6.4 

6.4 

3.8 

2.6 

14.43* 

0.1 12 

Year 5 

10.0 

14.6 

1 1.0 

9.4 

8.2 

3.8 

1 7.23*** 

0.1 14 

% Change from 
Yl to Y2 

0.0 

0.2 

0.0 

0.0 

0.5 

0.0 



% Change from 
Y2 to Y3 

4.4 

-0.2 

-3.7 

1.7 

0.2 

0.7 



% Change from 
Y3 to Y4 

-2.4 

1.8 

3.3 

3.1 

-4.0 

0.0 



% Change from 
Y4 to Y5 

5.6 

4.4 

4.6 

3.0 

4.4 

1.2 



Gap to the End-of- 
Project Target 

-9.0 

-4.4 

-8.0 

-9.6 

-10.8 

-17.8 




Note. Data for Year I were obtained from the Year I Report (Nixon et al., 20 1 2); Data for Year 2 were obtained from the 
Year 2 Report (Donahue et al., 20 1 2); and Data for Year 3 were obtained from the Year 3 Report (Donahue et al., 20 1 3). 
Effect size was calculated by using Cramer’s V (Ferguson, 2009): a value of 0.20 was the recommended minimum effect size 
representing a “practically” significant effect for social science data; a value of 0.50 presents a moderate effect size; and a 
value of 0.80 presents a strong effect size. 

*** p < 0.00 1 
** p < 0.0 1 

*p< 0.05 

Statistically significant school variations in the percentage of students meeting ACT 
benchmarks suggests that it is more appropriate to use multilevel modeling to examine the 
associations between student attributes and the percentages of students meeting or exceeding the 
benchmarks. Table 33 shows the results of the hierarchical generalized linear models. 

Table 33. Results of Hierarchical Generalized Linear Models: ACT Outcomes 


English Math Reading Science 

Fixed Effect 




OR 

ft 

OR 

P 

OR 

I 

OR 

Female a 

0.27’ 

1.31 

-0.40* 

0.67 

0.261 

1.30 

-0.34 

0.71 

Students with free meal 


0.44 


0.59 

-0.82 

0.44 


0.45 

k 

-0.82*** 

-0.52** 

-0.80*** 

status 

Students with reduced- 

-0.07 

0.93 

0.53 

1.70 

0.1 1 

l.l 1 

-0.02 

0.98 

price meal status c 
Hispanic d 

-0.73** 

0.48 

-1.02** 

0.36 

-0.64* 

0.53 

-0.78 

0.46 
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Fixed Effect 

English 

p OR 

Math 

P OR 

Reading 

P OR 

Science 

P OR 

Black e 

| | 

0.32 

- 1 .58*** 

0.21 

_iio*« 

0.33 

- 1 .20*** 

0.30 

Other Race f 

-0.24 

0.79 

0.28 

1.33 

0.22 

1.24 

-0.20 

0.82 


a Male is the reference group. 

b&c Students with paid meal status is the reference group, 
d, e, &f White students is the reference group. 

P= coefficient; OR = odds ratio 

The interpretation of odds ratio is that the odds of passing the benchmark is the degree to which the score deviates from I if 
the student was a “predictor = I (i.e., female, students with free meal status, students with reduced-price meal status, 
Hispanic, Black, and Other Race) in comparison to “predictor = 0 (i.e., male, students with paid meal status, and White).” If 
the odds ratio is greater than I, the predictor is more likely than the comparison to meet or exceed benchmarks; if the odds 
ratio is below I, the predictor is less likely than the comparison to meet or exceed benchmarks. 

***p < 0.00 1 
**p < 0.0 1 

*p < 0.05 

As shown in Table 33, results suggest that students’ college readiness skills measured by the 
ACT differed by student characteristics across the subject areas. Specifically, Black students, 
Hispanic students, and students who qualify for free meals (as compared to students with paid meal 
status) consistently have lower odds of meeting or exceeding the benchmarks in most subject areas. 
Males were found to be more college -ready than females in math while females were more college- 
ready than males in English. Findings for each subject area are discussed in detail below: 

• The odds of female students meeting or exceeding the English benchmark is 1.31 times 
greater compared to male students. In contrast, the odds of female students meeting or 
exceeding math benchmarks is 0.67 times lower compared to male students. 

• The odds that students who qualify for free meals have of meeting the English, math, 
and reading benchmarks are 0.44, 0.59, and 0.45 times lower as opposed to students with 
paid meal status. 

• Hispanic students consistently had lower performance levels than White students in 
relation to the English, math, and reading outcomes. The odds of meeting or exceeding 
the English, math, and reading benchmarks were 0.48, 0.36, and 0.53 times less likely for 
Hispanic students compared to White students. Similarly, Black students had lower 
performance levels than White students in all subject areas. The odds of meeting or 
exceeding the English, math, reading, and science benchmarks were 0.32, 0.21, 0.33, and 
0.30 times less likely for Black students as opposed to White students. 

Students’ Postsecondary Transition. JCPS provided data from the National Student 
Clearinghouse’s StudentTracker database to allow the evaluators to assess students’ postsecondary 
attendance. Based on 2010 data, 44% of the students who graduated in May of that year from the 
i3 project schools were enrolled in postsecondary institutions. The district mean for high schools 
for the same period was 55%; hence, the end-of-project target for the student transition outcome 
was set to 55%. 
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Table 34 shows changes in the postsecondary enrollment rates by project schools over the 
life of the grant through 2014. It should be noted that StudentTracker data are always one year 
behind the current project year so 2014 is the latest data point available. Although the majority of 
schools demonstrated an increase in the number of students attending postsecondary institutions 
between 2010 and 2012, this number decreased in 2013. However, the percentage increased at five 
out of the six i3 schools in 2014 with a 14% increase at Waggener High School. Overall, the Year 4 
average increased from Year 3 by 6.8% and from the baseline by 4%. The Year 4 enrollment rate 
was the second highest enrollment rate from all the project years, with only Year 2 having a higher 
percentage of enrolled students. However, the project did not meet the goal of 55%. 


Table 34. Postsecondary Enrollment Rates by i3 Project Schools Over Time 



2010 a 

201 1 a 

20l2 a 

2013 a 


2014 


School 

(Baseline) 

(Year 1) 

(Year 2) 

(Year 3) 


(Year 4) 


% 

% 

% 

% 

# 

# 

% 


Enrolled 

Enrolled 

Enrolled 

Enrolled 

Graduated 

Enrolled 

Enrolled 

Academy @ Shawnee 

34.0% 

33.0% 

46.1% 

29.3% 

89 

38 

42.7% 

Fern Creek Traditional 
High School 
Moore Traditional 
High School 
Valley Traditional 
High School 

56.8% 

53.6% 

60.3% 

54.2% 

300 

170 

56.7% 

37.3% 

47.1% 

63.0% 

44.4% 

153 

67 

43.8% 

28.2% 

31.1% 

45.6% 

27.0% 

193 

62 

32.1% 

Waggener High School 

55.4% 

41.8% 

58.6% 

35.3% 

137 

68 

49.6% 

Western High School 

36.4% 

41.4% 

36.1% 

39.5% 

151 

62 

41.1% 

Average 

Enrollment Rate b 

4 1 .4% 

41.3% 

51.6% 

38.3% 

1,023 

461 



a Enrollment rates for 2010 and 201 I for each project school were obtained from the Year 2 annual report (Donahue et at, 
2012); enrollment rates for 2012 were obtained from the Year 3 annual report (Donahue et at, 2013); and enrollment rates 
for 201 3 were obtained from the Year 4 annual report (Donahue et at, 2014). 

b The average enrollment rate was calculated as the average percentage enrollment rate across the six project schools, which 
accounted for variations in school size and the number of graduates among the i3 schools. 


As shown in Table 35, the majority of students who enrolled in public institutions in 2014 
(85%) was less than 1% greater than in 2013. However, the percentage of students enrolling in 
public institutions increased nearly 9% from the baseline year. In addition, the percentage of 
students enrolling in four-year institutions increased by 8% from Year 2 to Year 3. This percentage 
decreased between 2014 and 2010 by 3%, which indicates that more students seem to be enrolling in 
two-year institutions. This could be due in part to the added focus on two-year institutions and 
career pathways in the high schools. Finally, the percentage of students enrolling in Kentucky 
institutions decreased by 5% from Year 3 to Year 4; although, there was relatively no change 
between the baseline year and Year 4. 

The enrollment status of students who graduated from the JCPS i3 project schools in the 
spring of 2011, 2012, 2013, and 2014 and enrolled in postsecondary institutions in the fall of 2011, 
2012, 2013, and 2014, respectively, were also explored (see Table 36). In the fall of 2014, 38% of 
students enrolled full-time in postsecondary institutions, which was 7% lower than in 2013. The 
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percentages of students who enrolled full-time ranged from 31% (Valley High School) to 45% 
(Moore Traditional High School). Based upon preliminary postsecondary enrollment data, however, 
the enrollment status of approximately 52% of the May 2014 graduates from the i3 schools is 
unknown. 
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Table 35. Type of Institution that Students from i3 Schools Enrolled in Over Time 


School 

2010 

Public Institution 
2011 2012 2013 

2014 

2010 

Four-Year Institution 
2011 2012 2013 

2014 

2010 

Kentucky Institution 
2011 2012 2013 

2014 

Academy @ Shawnee 

72.2% 

96.9% 

9 1 .4% 

100.0% 

92.1% 

61.1% 

34.4% 

48.6% 

22.2% 

94.7% 

75.0% 

96.9% 

94.3% 

92.6% 

92.1% 

Fern Creek Traditional 
High School 

79.9% 

80.7% 

88.1% 

85.8% 

85.3% 

68.2% 

70.7% 

58.5% 

67.1% 

58.8% 

90.9% 

89.5% 

9 1 .5% 

94.2% 

90.6% 

Moore Traditional 
High School 

69.5% 

72.5% 

88.5% 

8 1 .6% 

74.6% 

64.4% 

60.0% 

50.6% 

59.2% 

62.7% 

88.1% 

90.0% 

89.7% 

94.7% 

88.1% 

Valley Traditional 
High School 

65.2% 

71.2% 

8 1 .9% 

83.0% 

88.7% 

52.2% 

54.2% 

55.6% 

44.7% 

50.0% 

91.3% 

83.1% 

9 1 .7% 

97.9% 

88.7% 

Waggener High School 

77.6% 

78.7% 

86.3% 

78.7% 

80.9% 

67.3% 

5 1 .7% 

45.1% 

63.9% 

55.9% 

9 1 .6% 

86.5% 

87.3% 

95.1% 

92.6% 

Western High School 

88.5% 

76.0% 

80.0% 

82.3% 

9 1 .9% 

5 1 .9% 

47.9% 

5 1 .4% 

58.1% 

43.5% 

94.2% 

89.6% 

97.1% 

95.2% 

85.5% 

Across the i3 Schools 

76.9% 

78.2% 



85.6% 



52.8% 

52.5% 

60.9% 




94.9% 

89.6% 


Table 36. Student Enrollment Status Over Time 


School 

Full-Time 

Part-Time 

Less than 
Part-Time 

Withdrawn 

Status was 
Not Defined 


2011 

2012 


2014 

2011 

2012 

2013 



2012 

2013 


2011 

2012 2013 

2014 

2011 

2012 2013 


Academy @ Shawnee 

37.5% 

40.0% 

18.5% 

36.8% 

- 

5.7% 

7.4% 

-- 

-- 

2.9% 

- 

- 

3.1% 

-- 

- 

7.9% 

59.4% 

51.4% 

74.1% 

55.3% 

Fern Creek Traditional 
High School 

63.5% 

48.3% 

55.3% 

42.5% 

4.4% 

6.8% 

5.3% 

2.4% 

-- 

1.1% 

2.0% 

1.8% 

2.2% 

1.1% 

2.0% 

4.7% 

29.8% 

42.6% 

35.5% 

48.8% 

Moore Traditional 
High School 

45.0% 

41.4% 

52.6% 

44.8% 

6.3% 

10.3% 

3.9% 

1.5% 

-- 

- 

1.3% 

1.5% 

8.8% 

2.3% 

39.5% 

4.5% 

40.0% 

46.0% 

2.6% 

47.8% 

Valley Traditional 
High School 

50.8% 

40.3% 

29.8% 

30.6% 

5.1% 

4.2% 

-- 

1.6% 

3.4% 

2.8% 

2.1% 

3.2% 

3.4% 

6.9% 

-- 

9.7% 

37.3% 

45.8% 

68.1% 

54.8% 

Waggener High School 

48.3% 

35.3% 

45.9% 

38.2% 

3.4% 

3.9% 

3.3% 

4.4% 

-- 

2.9% 

1.6% 

- 

1.1% 

2.0% 

6.6% 

4.4% 

47.2% 

55.9% 

42.6% 

52.9% 

Western High School 

42.7% 

41.4% 

35.5% 

35.5% 

7.3% 

1.4% 

4.8% 

-- 

2.1% 

-- 

-- 

4.8% 

3.1% 

5.7% 

8.1% 

8.1% 

44.8% 

51.4% 

51.6% 

51.6% 

i3 Schools 

51.6% 


45.4% 38.1% 

4.8% 

5.5% 



0.7% 

1.7% 


1.9% 

3.4% 

2.8% 

3.3% 

6.6% 

39.5% 

47.8%54.4% 
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Objective C: Students * Sense of Social Support from Adults 

Adult Social Support Scale. Five CSS items were selected as a measure of students’ 
perceptions of support from adults to replace the former CSCI scale. Of the valid student responses 
(« = 3,646), the mean of the Adult Social Support subscale was 2.98 (SD = 0.56). As illustrated in 
Figure 13, students reported feeling more positive than negative about adult social support items. 
The item “There is at least one adult at my school whom I feel I can trust” received the highest 
rating compared to all other items. 


There is at least one adult at my school who says 
positive things to me frequently. 

When I have a problem there is at least one adult 
at my school whom I can talk about my problem. 

There is at least one adult at my school whom 
I feel I can trust. 

I believe I can talk with my counselor or dean. 


I feel my teachers really care about me. 


I 


1 4 . 9 % 

85 . 1 % 


1 7 . 3 % 

82 . 7 % 


13 . 9 % 

86 . 1 % 


25 . 8 % 

74 . 2 % 


24 . 4 % 

75 . 6 % 


I 

0% 20% 40% 60% 80% 100% 


■ Negative ■ Positive 

Figure 1 3. Distribution of student responses for the Adult Social Support subscale. 

Note. Each CSS item was measured on a 4-point Likert scale (I = Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly 
agree), ratings of I and 2 were grouped as a negative response and ratings of 3 and 4 were grouped as a positive response. 


The examination of between-school differences on the Adult Social Support subscale 
suggested that, in Year 5, schools varied in their perceived levels of adult social support, 

F( 5, 4631) = 10.80,p> < 0.001, rf = 0.012. In particular, students at Fern Creek Traditional High 
School perceived a significantly higher level of adult social support in comparison to students from 
other high schools, except Waggener High School. 


Subgroup analyses were conducted to assess whether students’ demographic characteristics 
moderated student ratings of adult social support. The demographics included gender, grade level, 
race/ ethnicity, FRPM status, and LEP status. Findings are summarized as follows: 


• Female students reported higher levels of adult social support than did male students, 
<4635) = 2.38, A = 0.018, rf = 0.001. 

• Student ratings of adult social support differed by grade level, F( 3, 4620) = 8.39, 

p < 0.001, r] 2 = 0.005. Specifically, 1 1th and 12th graders reported a higher level of adult 
social support than did ninth and 10th graders (p < 0.01). 
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• Student ratings of adult social support differed by race, F( 4, 4632) = 3.53, p — 0.007, 
yf — 0.003. Specifically, Asian students reported a higher level of adult social support 
than did Black students (p = 0.011). 

• Student ratings of adult social support differed by their FRPM status, F( 2, 4634) = 3.39, 
p = 0.34, rf = 0.001 . Specifically, students with paid meal status reported a higher level 
of adult social support than did students receiving free meals (p — 0.05). 

• Student ratings of adult social support differed by their LEP status, F(l, 4635) = 9.77, 
p = 0.002, y] 2 = 0.002. Specifically, students with LEP reported a higher level of adult 
social support than did students without LEP. 

Peer Social Support Scale. One item, “I really like other students in my school,” was 
used as an indicator of the sense of social support among students. Of the valid student responses 
(» = 4,634), the mean Peer Social Support item score was 2.59 (SD= 0.79). As illustrated in 
Figure 14, students reported feeling more positive than negative about peer social support. 


1 really like other students in my school. 


39 . 7 % 

60 . 3 % 




■ Negative H Positive 

0% 

20% 

40% 60% 


Figure 14. Distribution of student responses for the Peer Social Support subscale. 

Note. Each CSS item was measured on a 4-point Likert scale (I = Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly 
agree), ratings of I and 2 were grouped as a negative response and ratings of 3 and 4 were grouped as a positive response. 

The examination of between-school differences on the Peer Social Support item suggested 
that, in Year 5, schools varied in their perceived levels of peer social support, F( 5, 4628) = 3.78, 
p — 0.002, yf = 0.004. In particular, students at Fern Creek Traditional Fligh School perceived 
significantly higher levels of peer social support in comparison to students from Moore Traditional 
School (p — 0.001) and Western Fligh School (p = 0.44). 

Subgroup analyses were conducted to assess whether students’ demographic characteristics 
moderated student ratings on the Peer Social Support subscale. The demographics included gender, 
grade level, race/ethnicity, FRPM status, and LEP status. Findings are summarized as follows: 

• Male students reported a higher level of peer social support than did female students, 
F(l, 4632) = 130.85 , p < 0.001, r] 2 = 0.027. 

• Students’ perceptions of peer social support differed by their grade level, 

F(3, 4617) = 3. 80, A = 0.01, yf = 0.002. In particular, ninth graders reported a higher 
level of peer social support than did 10th graders (p = 0.023). 
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• Student ratings of peer social support differed by race, F( 4, 4629) = 5.82 ,p < 0.001, 
rf — 0.005. Specifically, Asian students reported a higher level of peer social support 
than did students from all other racial/ethnic backgrounds (p < 0.01). 

• Students’ perceptions of peer social support did not differ by their FRPM status, 

F(2, 4631) = 0.297 ,p = 0.743, rf = 0.000. 

• Students with LEP reported a higher level of peer social support than did students 
without LEP, F(l, 4632) = 20.08,/> < 0.001, rf = 0.004. 

Objective E: School Connectedness and Engagement 

School Connectedness and Engagement Scale. Five CSS items were selected as a 
measure of students’ perceptions of school connectedness and engagement to replace the CSCI 
items. It should be noted that, of the five items, two items related to school engagement were 
measured as “Yes/No” items (see Figure 15, items B17 and B18). Overall, only about a third of the 
students were involved in some sort of club or sports team. In terms of items related to students’ 
perceptions of school connectedness, a mean score was calculated based on three items measured 
on a 4-point Likert scale (see Figure 15, items B3, B5, and B23). Of the valid student responses 
(» = 4,643), the scale mean was 2.64 ( SD = 0.65). As illustrated in Figure 15, students reported 
feeling more positive than negative about school connectedness items. 


1 participate in clubs or activities (besides 
sports) sponsored by my school (B 1 8). 

1 participate on sports teams sponsored 
by my school (B 1 7). 

62 . 8 % 

37 . 2 % 



63 . 5 % 

36 . 5 % 




1 am very satisfied with my school (B23). 

35 . 2 % 

64 . 8 % 




1 enjoy going to school (B3). 

42 . 4 % 

57 . 6 % 




1 feel that 1 belong in my school (B5). 

32 . 2 % 

67 . 8 % 


0% 20% 40% 60% 80% 100% 


■ Negative ■ Positive 

Figure 1 5. Distribution of student responses for the 
School Connectedness and Engagement subscale. 

Note. Items B 1 7 and B 1 8 were measured by “Yes/No” responses. Items B3, B5, and B23 were measured on a 4-point Likert 
scale (I = Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly agree), ratings of I and 2 were grouped as a negative 
response and ratings of 3 and 4 were grouped as a positive response. 

The examination of between-school differences suggested that, in Year 5, schools varied in 
students’ perceptions of school connectedness, F(5, 4637) = 17.89 ,p < 0.001, rf = 0.019. In 
particular, students at Fern Creek Traditional High School perceived significantly higher levels of 
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school connectedness in comparison to students from other Moore Traditional Schools (p < 0.001), 
Waggener High School (p = 0.027), and Western High School (p < 0.001). Students at Valley 
Traditional High School also reported a higher level of school connectedness than did students at 
Moore Traditional School (p < 0.001) and Western High School (p < 0.001). Students at Moore 
Traditional School reported the lowest level of school connectedness as compared to all other 
schools, except Western High School (p < 0.01). 

Subgroup analyses were conducted to assess whether students’ demographic characteristics 
moderated their ratings on the School Connectedness and Engagement subscale. The demographics 
included gender, grade level, race/ ethnicity, FRPM status, and LEP status. Findings are summarized 
as follows: 


• Male students reported a higher level of school connectedness than did female students, 
F(l, 4641) = 18.63 , p < 0.001, rf = 0.004. 

• Students’ perceptions of school connectedness differed by student grade level, 

F( 3, 4626) = 3.48, p — 0.015, rf — 0.002. In particular, 11th graders reported a higher 
level of school connectedness than did 10th graders (p = 0.019). 

• Student ratings of school connectedness and engagement differed by race, 

F’(4, 4638) = 10.61 ,p < 0.001, rf = 0.009. Specifically, Asian students reported the 
highest level of school connectedness, followed by Hispanic students (p < 0.05). 

• Students’ perceptions of school connectedness and engagement did not differ by 
students’ FRPM status, F( 2, 4640) = 1.55 ,p = 0.212, rf — 0.001. 

• Students with LEP reported a higher level of school connectedness and engagement 
than did students without LEP, F(l, 4641) = 45. 03, A < 0.001, rf = 0.010. 

College and Career Readiness. In Year 5, McREL evaluators also requested nine CSS 
items related to students’ perceptions of college readiness (see Figure 16). The reliability for the 
scale was 0.76. In this section, evaluators examine the extent to which student perceptions of 
college readiness differ by school and demographic characteristics (i.e., gender, grade level, race, 
FRPM status, and LEP status). A College Readiness subscale mean score was calculated and used 
for the analysis. Of the valid student responses (« = 4,627), the scale mean was 3.12 (SD = 0.50). 
As illustrated in Figure 16, students reported feeling more positive than negative about their college 
readiness. 
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I can make more money if I have a college degree. 


8 . 5 % 


91 . 5 % 


My parents expect me to go to college. 

My counselor has talked with me about my future 
after high school with college as the goal. 

My teachers talk about college issues, like 
requirements and majors. 

I feel my teachers believe I can succeed in college. 


I plan to go to college. 

I believe I am developing essential skills for life 
(such as reading, writing, and math) in JCPS. 

My school does a good job of preparing me for 
college. 

My JCPS education will prepare me for 
employment. 


I 1.1% 


88 . 9 % 


33 . 0 % 


18 . 1 % 


1 2 . 9 % 


1 2 . 8 % 


16 . 4 % 


22 . 3 % 


22 . 2 % 


0 % 


20 % 


67 . 0 % 


81 . 9 % 


87 . 1 % 


87 . 2 % 


83 . 6 % 


77 . 7 % 


77 . 8 % 


40% 60% 80% 

■ Negative ■ Positive 


100 % 


Figure 1 6. Distribution of student responses for the College Readiness subscale. 

Note. Each CSS item was measured on a 4-point Likert scale (I = Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly 
agree), ratings of I and 2 were grouped as a negative response and ratings of 3 and 4 were grouped as a positive response. 

The examination of between-school differences suggested that, in Year 5, schools varied in 
students’ perceived levels of school connectedness, F( 5, 4621) = 12.16, p < 0.001, rf = 0.014. 
Subgroup analyses were conducted to assess whether students’ demographic characteristics 
moderated their ratings on the College Readiness subscale. The demographics included gender, 
grade level, race/ethnicity, FRPM status, and LEP status. Findings are summarized as follows: 

• Female students reported a higher level of college readiness than did male students, 

F(l, 4625) = 23.12, fl < 0.001, rf = 0.005. 

• Students’ perceptions of college readiness differed by student grade level, 

F( 3, 4612) = 5.02, p = 0.002, rf = 0.003. In particular, 10th graders reported the lowest 
level of college readiness than did ninth, 1 1th, and 12th graders (p < 0.05). 

• Student ratings of college readiness differed by race, F(4, 4622) = 9.91 ,p < 0.001, 

rf = 0.009. Specifically, Asian students reported a higher level of college readiness than 
did White and Hispanic students (p < 0.05); Black students also reported a higher level 
of college readiness than did White students (p < 0.001). 
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• Students’ perceptions of college readiness did not differ by their FRPM status, 

F(2, 4624) = 1.28, p = 0.279, rf = 0.001. 

• Students with LEP reported a higher level of college readiness than did students without 
LEP, F(l, 4625) = 12.79 ,p < 0.001, rf = 0.003. 

Implementation Fidelity for CAT 

There are seven indicators for this the CAT measure. The indicators for this construct 

follow: 


• CARTs are assigned to each school. 

• CARTs receive professional development. 

• CARTs develop curriculum topics in a timely manner. 

• CARTs develop and deliver CAT lessons in a timely manner. 

• CARTs use a walkthrough tool to monitor CAT advisory periods. 

• CARTs provide training to advisors. 

• CAT advisory periods are offered frequently. 

Each project school was scored individually on these indicators. The scores were aggregated 
across the schools for one overall fidelity score. Table 37 shows the full fidelity table and scores. 
Due to a lack of data, assumptions were made on the fidelity rubric for some of the schools based 
on informal conversations or e-mails as well as past history. None of the i3 project schools 
implemented the CART components with a score higher than 12. In order to reach a threshold 
indicating that the schools implemented the component with fidelity, 67% of the schools needed to 
reach a score of 1 2 or higher; thus, this component was not implemented with fidelity. 
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Table 37. JCPS Fidelity Measure and Scores for Component 2: CAT 


Key Elements of 
Component 

Operational 
Definition 
for Indicator 

Data 

Source(s) for 
Measuring 
the Indicator 

Data 

Collection 
Schedule for 
the Indicator 

Raw 

Possible 
Indicator 
Score Per 
School 

Scoring and Scoring Criteria for Fidelity 

Scoring Indicator 

Criteria School Name Score 

1 

College Access 
Resource 

Each school has a 
CART assigned 

District and 
school records 

Records 
collected by the 

0-1 

0 = Low 

CART is not assigned 

Academy @ 
Shawnee 

1 


Teacher (CART) 
assigned to each 
school 


evaluation team 
annually 


1 = High 

Fern Creek 

1 





CART is assigned 

Moore 

1 








Valley 

1 








Waggener 

1 








Western 

1 

2 

CART receives 
professional 

CARTs participate 
in CAT 

CART Survey 3 

Survey 

administered by 

0-3 

0 = Low 

Attended no sessions 

Academy @ 
Shawnee 

1 


development 

professional 
development (one 
CART per school) 


the evaluation 
team annually 


1 = Adequate 

Fern Creek 

1 





Attended 1-2 sessions 

2 = Moderate 
Attended 3-4 sessions 

Moore 

0 







Valley 

0 







3 = High 

Attended *> nr more 

Waggener 

0 







sessions 

Western 

0 

3 

CARTs develop 
and deliver CAT 

Timely availability 
of lessons to 

CART and 
Teacher 

Surveys 

administered by 

0-3 

0 = Low 

Teachers do not receive 

Academy @ 
Shawnee 

3 


lessons in a 
timely manner 

teachers 

Surveys 

the evaluation 
team annually 


lessons 
1 = Adequate 

Fern Creek 

3 







Teachers receive lessons 
on the morning of 

Moore 

3 







advisory 

Valley 

3 
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Data Data 


Operational Source(s) for Collection 
Key Elements of Definition Measuring Schedule for 
Component for Indicator the Indicator the Indicator 







4 

CARTs use a 
walkthrough tool 
to monitor CAT 
advisory periods 

CARTs use the 
walkthrough tool 
to monitor 
advisories 

CART Survey 3 

Survey 

administered by 
the evaluation 
team annually 
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Raw Scoring and Scoring Criteria for Fidelity 


Possible 

Indicator 

Score Per Scoring Indicator 


School Criteria School Name Score 


2 = Moderate 
Teachers receive lessons 
one day before advisory 


3 = High 


0-4 


Teachers receive lessons 
two or more days before 
advisory 

0 = Low 

CARTs use walkthroughs 
less than once a 
trimester 


I = Adequate 
CARTs use walkthroughs 
once a trimester 


2 = Moderate 
CARTs use walkthroughs 

monthly 

3 = Sufficient 
CARTs use walkthroughs 

biweekly 

4 = High 

CARTs use walkthroughs 
weekly 


Waggener 

3 

Western 

3 

Academy @ 
Shawnee 

2 

Fern Creek 

1 

Moore 

0 

Valley 

0 

Waggener 

0 

Western 

0 



Data Data 


Operational Source(s) for Collection 
Key Elements of Definition Measuring Schedule for 
Component for Indicator the Indicator the Indicator 


5 

CARTs provide 
training to 
advisors 

The number of 
hours that CARTs 
provide training to 
teachers on how 
to be effective 
advisors 

CART and 

Teacher 

Surveys 

Surveys 

administered by 
the evaluation 
team annually 

6 

CAT advisory 
periods are 
offered 
frequently 

How often CAT 
advisory periods 
are provided at 
the school 

CART and 

Teacher 

Surveys 

Survey 

administered by 
the evaluation 
team annually 
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Raw Scoring and Scoring Criteria for Fidelity 


Possible 
Indicator 
Score Per 
School 

Scoring 

Criteria 

0-3 

0 = Low 

Zero hours of training 
with teachers 


1 = Adequate 
1-2 hours of training 
with teachers 


2 = Moderate 
2-3 hours of training 
with teachers 


3 = High 

More than 3 hours of 
training with teachers 

1-3 

1 = Low 
Bimonthly 


2 = Moderate 
Weekly 


3 = High 
Twice a Week 


Indicator 

School Name Score 


Academy @ 
Shawnee 

1 

Fern Creek 

1 

Moore 

1 

Valley 

1 

Waggener 

1 

Western 

1 

Academy @ 
Shawnee 

1 

Fern Creek 

1 

Moore 

1 

Valley 

2 

Waggener 

2 

Western 

2 



Data Data 

Operational Source(s) for Collection 

Key Elements of Definition Measuring Schedule for 

Component for Indicator the Indicator the Indicator 

Raw Scoring and Scoring Criteria for Fidelity 

Possible 

Indicator 

Score Per Scoring Indicator 

School Criteria School Name Score 



12-17 = 

High Implementation 

Academy @ 
Shawnee 

9 




Fern Creek 

8 

Component Level Score 

0-17 

6-11 = 
Moderate 

Moore 

6 



Implementation 

Valley 

7 



0-5 = 

Waggener 

7 



Low Implementation 

Western 

7 

Implementation with fidelity = At least 67 % of the schools have a minimum score of 12 or above 
Implementation without fidelity = Fewer than 67% of the schools have a score of 12 or above 

No school has a score of more 
than 12; thus, this component 
has not been implemented with 
fidelity 


a Only four CARTs turned in surveys, as they were collected anonymously, these totals are being assigned to the first schools on the list. The project director indicated that 
professional development sessions did not happen frequently this year so no school would score above a “I,” even if they had returned the survey. 
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Project Goal 3: Teachers’ Pedagogical and Student Support Practices 


The third project goal is to improve teachers’ pedagogical and student support practices to 
maximize the effectiveness of increased learning time. Strategies to reach this goal include ensuring 
that content-based and cross-disciplinary PLCs meet regularly during the school year. Measures to 
indicate progress on this goal include changes in teachers’ perceptions of their ability to identify and 
appropriately respond to unique academic and social needs of students as defined by the 
2015 Teacher Survey. Table 38 outlines the objectives for Project Goal 3 and the yearly or end-of- 
project target, when appropriate. 


Table 38. Project Goal 3 Objectives, Performance Measures, and Targets 


Objective 

Performance 

Measure 

T arget 

A 

Ensure that content-based and cross-disciplinary 
professional learning communities (PLCs) meet 
regularly over the school year. 

Frequency of 
content-based PLCs 
in core courses 

1 5 content-based 
PLCs annually 

Total time for 
content-based PLCs 

60 hours 

Frequency of cross- 
disciplinary PLCs 

32 cross-disciplinary 
PLCs annually 

Total time for cross- 
disciplinary PLCs 

29 hours 

B 

Increase teachers’ perceptions of collaboration among 
themselves within and across content areas in the 
school 

Collaboration Scale 
on the Teacher 
Survey 

30.38 a 

C 

Increase teachers’ perceptions of their own self- 
efficacy with respect to content-based knowledge for 
teaching 

Self-Efficacy Scale on 
the Teacher Survey 

33.77 b 

D 

Improve teachers’ perceptions of their instructional 
practices in order to increase student academic 
engagement and challenge 

Teaching Scale on 
the CSS 

1 0.00 c 

E 

Improve teachers’ perceptions of their ability to 
identify and appropriately respond to students’ unique 
academic and social needs 

CSCI Support for 
Learning and Adult 
Social Support scales 

4.04 and 4.22, 
respectively 


a The reported 20 1 3 mean and standard deviation for the Teacher Collaboration scale on the Teacher Survey was 28.20 and 

4.36, respectively. The target for Years 4 and 5 was set as the growth of half a standard deviation, or 30.38. 

b The reported 20 1 3 mean and standard deviation for the Self-Efficacy scale on the Teacher Survey was 3 1 .59 and 4.35, 

respectively. The target for Years 4 and 5 was set as the growth of half a standard deviation, or 33.77. 

c The reported 201 3 mean and standard deviation for the Teaching scale on the CSS was 8.75 and 2.50, respectively. The 

target on Years 4 and 5 was set as the growth of a half a standard deviation, or 1 0.00. 

Objective A: Professional Learning Community (PLC) Meetings 

Project staff commented that this was arguably the most successful i3 intervention because it 
integrated a culture of teacher collaboration among the schools that will continue beyond the life of 
the grant. A large part of that success is the “homegrown assessment” system where all teachers in a 
school who teach a common course, such as English I, can have a conversation with colleagues 
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about the students enrolled. Additionally, through this intervention, JCPS adapted the 
PLC framework and supported training for both teachers and principals. 

An interviewed principal explained that there are three sets of questions that his school tries 
to respond to and discuss during a PLC meeting: 

1. What does our current progress data tell us? How do we celebrate? 

2. In terms of planning, what’s working and what’s not working? Do we plan a school 
remediation day or continue on to the next unit? 

3. How do we overcome any obstacles we may be facing in the classroom? 

This is also a time for teachers to discuss individual students and what problems they may be facing. 
For example, if a student has been absent for multiple days, how can the team address the problem? 

Content-Based PLCs. The content-based PLC target of 60 hours was based on an 
original estimate of having 15 four-hour-long, half-day sessions. The target for the content-based 
PLCs was based on the i3 schools having 55-minute PLC periods where 29 total hours would be 
devoted to the PLCs. Although some teachers may attend half-day training sessions (e.g., DuFour 
PLC training), individual schools tend to hold PLC meetings regularly within their buildings and 
each school conducts these PLCs in a different way. In order to report consistently across the 
schools, Teacher Survey results will be used to estimate how well the schools are doing in reaching 
the objectives’ targets and to calculate the fidelity implementation score. 

All but four teachers indicated that they participated in at least one content-based PLC in 
their school. Assuming there are 34 weeks in a school year, Table 39 estimates the average 
frequency and duration of the content-based PLCs at each i3 project school. Based on these 
estimates, the i3 schools exceeded the target of having 15 content-based PLC meetings per year. 
Typically, most of the schools have meetings on a weekly basis; therefore, they are meeting an 
average of 34 times per year. In addition, four out of the six i3 schools meet for 60 minutes per 
session. Because two schools do not meet for a full hour, the project average is just below the 
60-minute target. 


Table 39. Average Frequency and Duration of Content-Based PLCs by School 


School 

Meetings 
Per Week 

Frequency 
(# Per Year) 

Minutes 
Per Meeting 

Duration 
(Hours Per 
Year) 

Academy @ Shawnee 

1 

34 

60 

34 

Fern Creek Traditional High School 

1 

34 

60 

34 

Moore Traditional High School 

1 

34 

30 

17 

Valley Traditional High School 

1 

34 

45 

26 

Waggener High School 

1 

34 

60 

34 

Western High School 

1 

34 

60 

34 

Across the i3 Schools 

1 

34 

52.5 

29.8 
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Principals were also asked to comment on factors that facilitated content -based PLC 
implementation in their schools and challenges they faced. Table 40 lists those successes and 
challenges. 


Table 40. Principal Reports of Facilitating and Challenging Factors of Content-Based PLC 
Implementation 


Success Factors 

Challenges 

Alignment of standards — all teachers are on the 
same page 

Buy-in for some 

Common curriculum 

Most groups have common planning [periods]; 
a few do not. Those that do not have to meet 
outside the school day 

Weekly common formative assessment data 

Small teams of teachers 

Common assessments 

AP versus regular curriculum 

Differentiation based on formative data 

Time for PLC work is a challenge; we have 
incorporated planning sessions for six weeks each to 
give teachers more time to work together 

Synergy — the group is stronger than an individual 
and the group benefits from a variety of ideas to 
enhance instruction 


Each group reports the percent of students 
proficient in each standard throughout the year. 
Year-end tests are not a surprise; each PLC knows 
where students are with each standard. 

Continue to support school with i3 funds; these are 
used to create time for teachers to meet 

Improved interventions 



Cross-Disciplinary PLCs. A cross-disciplinary PLC can be defined as a School of Study, 
Math-English-Social Studies-Science (MESS) group, or freshmen academy, among others. When 
asked if they participated in a cross-disciplinary PLC, only 40% of teachers responded that they did. 
Table 41 estimates the extent to which the 40% of teachers who reported that they participated in a 
cross-disciplinary PLC have met the project’s targets. For the cross-disciplinary PLCs, neither the 
frequency nor the duration targets have been met (M = 10.8 and M = 7.7, respectively). This is well 
below the goal of having 32 cross-disciplinary PLC meetings per year, totaling 29 hours. Many 
teachers chose to skip these questions or indicated that they did not know. These responses were not 
included in the analysis, but it does indicate that many teachers may not be aware that this work is 
happening within the schools. 


Table 41. Average Frequency and Duration of Cross-Disciplinary PLCs by School 


School 

Frequency of 
Meetings 

Frequency 
(# Per Year) 

Minutes 
Per Meeting 

Duration 
(Hours Per 
Year) 

Academy @ Shawnee 

Once or twice 
a year 

2 

60 

2 

Fern Creek Traditional High School 

Monthly 

9 

60 

9 

Moore Traditional High School 

Weekly 

34 

30 

17 

Valley Traditional High School 

Monthly 

9 

45 

7 
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School 

Frequency of 
Meetings 

Frequency 
(# Per Year) 

Minutes 
Per Meeting 

Duration 
(Hours Per 
Year) 


Waggener High School 
Western High School 


Monthly 

Once or twice a 
year 


9 

2 


60 

60 


9 

2 


Across the i3 Schools 

- 


52.5 

7.7 


Principals were also asked to comment on factors that facilitated cross-disciplinary PLC 
implementation in their schools and the challenges they faced. Table 42 lists those successes and 
challenges. 


Table 42. Principal Reports of Facilitating and Challenging Factors of Cross-Disciplinary 
PLC Implementation 


Success Factors 

Challenges 

Authentic assessments 

Cohort scheduling 

Differentiation for age-specific needs 

Finding time to plan is a challenge 

Students see connections between content 

Time during the day due to content PLCs 

Authentic lessons 

Finding the right teachers to pair 

Relevance and real-world applications across 
subjects 

Students achieve more and have better results on 
tests 

Teachers from different content areas work 
together and think outside the box 

Students are not in pure cohorts 


Objective B: Increase Teachers ’ Perceptions of Collaboration Among Teachers 
Within and Across Content Areas 

The Teacher Collaboration and Support scale, consisting of nine items, measures teachers’ 
perceptions of the levels of collaboration and support among teachers at their school. Items in this 
section follow: 

• Teachers have sufficient time to collaborate with colleagues to plan and improve 
instruction. 

• Collaboration is viewed as an important part of this school’s culture. 

• Teachers have sufficient resources to provide quality instmction. 

• Teachers are generally willing to try new ideas. 

• Teachers in this school trust each other. 

• It is okay in this school to discuss with other teachers positive and/ or negative feelings, 
concerns, and frustrations related to instruction and practice. 

• Teachers discuss difficulties and work together to develop solutions. 
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• Teachers support each other’s efforts to make improvements. 

• Access to expertise and resources is available and provided in a timely manner. 

The internal consistency reliability (Cronbach’s a) estimate of the Teacher Collaboration and 
Support scale was 0.83. Respondents selected answer choices based on a 4-point Likert scale 
(1 = Strongly disagree , 2 = Disagree, 3 = Agree, and 4 = Strongly agree). The scales were summed to 
generate a scale score. 

Across the core content teachers at JCPS i3 schools (n — 94), teachers reported high levels of 
teacher collaboration and support (M = 27.00, SD = 3.69). Figure 17 displays the percentage 
average across schools of teachers with negative (i.e., strongly disagree or disagree ) and positive (i.e., agree 
or strongly agree) responses on the Teacher Collaboration and Support scale. The project target for 
this scale was M = 30.38, which was not met by any of the i3 schools. 


16.4% 83.6% 


■ Negative ■ Positive 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

Figure 1 7. Distribution of teacher responses for the 
Teacher Collaboration and Support scale. 

Nearly 84% had positive responses across the scale. Table 43 lists the items as well as the 
teachers’ level of agreement. Thematically, the items that tended to have higher levels of agreement 
focused on teachers supporting one another, collaborating, and working together. Collaboration is 
perceived as part of the school’s culture, and teachers seem to be supporting each other and working 
well together to solve problems. This may be attributable to the PLCs, where time is devoted for 
teachers to work collaboratively to discuss students’ progress and challenges in a supportive work 
environment. 


Table 43. Agreement with Teacher Collaboration and Support Scale Items 


Item 

Aggregated Percentage 
of Positive Responses 
(Agree or Strongly Agree) 

Teachers are willing to try new ideas. 

94.7% 

Collaboration is seen as an important part of this school’s culture. 

93.6% 

Teachers support each other’s efforts to make improvements. 

89.3% 

It is okay in this school to discuss with other teachers positive and/or negative 
feelings, concerns, and frustrations related to instruction and practice. 

83.0% 

Teachers discuss difficulties and work together to develop solutions. 

8 1 .9% 

Teachers have sufficient time to collaborate with colleagues to plan and 
improve instruction. 

78.7% 
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Aggregated Percentage 

Item 

of Positive Responses 
(Agree or Strongly Agree) 

Access to expertise and resources is available and provided in a timely manner. 

77.7% 

Teachers in this school trust each other. 

76.6% 

Teachers have sufficient resources to provide quality instruction. 

76.6% 


Objective C: Increase Teachers * Perceptions of Their Own Self-Efficacy with 
Respect to Content-Based Knowledge for Teaching 

The Teacher Self-Efficacy scale, consisting of 10 items, measures teachers’ perceptions of 
their own self-efficacy. Items in this section follow: 

• How often do you feel that you are able to work effectively? 

• How often are you satisfied with the quality of your work? 

• How often do you feel that you are being successful in your work? 

• How often do you have sufficient self-confidence to defend your own points of view? 

• How often do you feel adequately prepared to use formative data (e.g., class work, 

homework, and other instructional activities) to adjust your instruction? 

• How often do you feel adequately prepared to use summative data (e.g., end of chapter, 
unit, or course assessments) to adjust your instruction? 

• How often do you feel confident in teaching all required content standards in your 
subject area? 

• How often do you feel confident in adjusting your instructional practices to meet the 
needs of individual students? 

• How often do you worry about being criticized if positive results are not readily 
apparent? 

• How often does the emphasis on success discourage you from trying new approaches? 

The internal consistency reliability (Cronbach’s a) estimate of the Teacher Self-Efficacy scale 
was 0.67. Respondents selected frequency answer choices based on a 4-point Likert scale (1 = Neper, 
2 = Sometimes , 3 = Often, and 4 = Very often), with a summed scale range of 10 to 40 points. 

Across the core content teachers at JCPS i3 schools (n — 87), teachers reported moderately 
high levels of teacher self-efficacy (M = 31.32, SD = 16.21) and did not surpass the end-of-project 
target of 33.77. Figure 18 displays the percentage average across the schools of teachers who 
responded to each category on the Teacher Self-Efficacy scale. Overall, the teachers in the six 
i3 project schools seem to have high perceptions of their own self-efficacy as 89% of them provided 
positive responses. 
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100 % 


75 % 

51 . 3 % 

50 % 

25 % 

0 % 

Never Seldom Sometimes Almost Always 

Figure 18. Distribution of teacher responses for the Teacher Self-Efficacy scale. 

Survey responses also indicated that over half of the teachers are reluctant to try new ideas 
because of the emphasis on success. According to the survey, 59% of the core content teachers are 
sometimes or almost always discouraged from trying new approaches because of the emphasis on 
success. A high percentage of teachers frequently choose not to try new approaches because of the 
pressure of achieving high results. 

Objective D: Improve Teachers * Perceptions of Their Instructional Practices in 
Order to Increase Student Academic Engagement and Challenge 

CSS Teaching Scale 

JCPS provides staff members who have been CSS-certified with data for the i3 project 
schools to gauge teachers’ perceptions of their instructional practices in order to increase students’ 
academic engagement and challenge. Three items from the CSS relate to this construct: 

• If I try really hard, I can get through to even the most unmotivated students. 

• By trying a different teaching method, I can significantly affect a student’s achievement. 

• If teachers have willingness, they can help any student learn. 

The internal consistency reliability (Cronbach’s a) estimate of the CSS Teaching scale was 
0.83. Respondents selected answer choices based on a 4-point Likert scale (1 = Strongly disagree , 

2 = Disagree, 3 = Agree, and 4 = Strongly agree with an N/A choice). 

Across the core content teachers at JCPS i3 schools (» = 491), teachers reported moderately 
high levels of student academic engagement (M = 8.88, SD = 2.18), although it did not meet the 
end-of-project target of 10.00. Table 44 shows the items and the teachers’ level of agreement. 
Certified staff in the project schools tended to have high levels of agreement with all three items. 
Specifically, 9 1 % of the respondents agreed or strongly agreed that by trying different teaching 
methods, they can significantly affect student achievement. This indicates that teachers attempt to 



80 


meet individual learning needs for all students. Over 70% of the teachers strongly agreed or agreed 
with the other two items. 


Table 44. Teachers’ Perceptions of Their Instructional Practices to Increase Student 
Academic Engagement and Challenge 


Items 

Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

N/A 

If 1 try really hard, 1 can get through to even the 
most unmotivated students. 

2.8% 

6.1% 

20.7% 

49.8% 

20.5% 

By trying a different teaching method, 1 can 
significantly affect a student’s achievement. 

1.0% 

4.7% 

60.4% 

28.7% 

5.3% 

If teachers have willingness, they can help any 
student learn. 

1.4% 

3.3% 

15.0% 

50.8% 

29.5% 


Student Academic Engagement. Teachers’ perceptions of student academic 
engagement and challenge were also measured through the 2015 Teacher Survey. Consisting of six 
items, this scale measures teachers’ perceptions of the extent that students are academically engaged 
in their school. Items in this section follow: 

• Teachers at my school challenge students academically. 

• Teachers at my school encourage students to go beyond stated expectations. 

• Teachers at my school involve students in setting expectations. 

• Teachers at my school involve students in planning lessons. 

• Teachers at my school involve students in developing criteria for assessing their 

assignments. 

• Teachers at my school work with disenfranchised students to help them feel more 
connected to school. 

The internal consistency reliability (Cronbach’s a) estimate of the Student Academic 
Engagement scale was 0.81. Respondents selected answer choices based on a 4-point Likert scale 
(1 = Strongly disagree , 2 = Disagree, 3 = Agree, and 4 = Strongly agree). 

Across the core content teachers at JCPS i3 schools {n — 875), teachers reported moderately 
high levels of student academic engagement (M = 16.35, SD = 3.04). Figure 19 displays the 
percentage average across the schools of teachers who had negative (i.e., strongly disagree or disagree) or 
positive (i.e., strongly agree or agree) responses on the Student Academic Engagement scale. Overall, 
almost two thirds of the respondents demonstrated positive agreement on the Student Academic 
Engagement scale. The percentages of teachers responding with a rating of agree or strongly agree was 
more than 80% on two items: “Teachers at my school challenge students academically” (81%) and 
“Teachers at my school work with disenfranchised students to help them feel more connected to 
school” (86%). 
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36 . 6 % 


63 . 4 % 


■ Negative ■ Positive 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

Figure 1 9. Distribution of teacher responses for the Student Academic Engagement scale. 

Items regarding how teachers involve students in planning tended to have more negative 
responses. For example, nearly three quarters of the teacher respondents indicated that they strongly 
disagree or disagree with the item, “Teachers at my school involve students in planning lessons” (76%). 
Similarly, 56% of teachers strongly disagreed or disagreed that “Teachers at my school involve 
students in developing criteria for assessing their assignments.” 

Thus, it seems that teachers tended to rate items that focused on challenging students and 
working with disenfranchised students higher than items related to students being actively involved 
in planning lessons, setting expectations, and developing criteria for lessons. It appears that teachers 
have fewer tendencies to actively engage students in planning lessons, but do perceive themselves to 
be challenging students academically and setting high expectations for them. 

Objective E: Improve Teachers ’ Perceptions of Their Ability to Identify and 
Appropriately Respond to Student’s Unique Academic and Social Needs 

The last objective for Project Goal 3 is to measure teachers’ perceptions of their ability to 
respond to students’ unique academic, social, and emotional needs. This objective is measured 
through two scales on the CSS: Support for Learning and Adult Social Support. 

Support for Learning Scale. Six CSS items were selected to assess school personnel’s 
perceptions of support for learning for their students. Of the valid student responses (n = 492), the 
scale mean was 2.94 (SD = 0.61). As illustrated in Figure 20, school personnel reported feeling 
more positive than negative about the support for learning items. However, one item received a 
much lower rating as compared to all other items, “Teachers at my school assign meaningful 
homework on a regular basis” (55% positive responses). 
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My classroom assessment feedback to students is 
frequent, descriptive, constructive, and immediate. 


At my school, teachers talk to students about 
their school progress. 


At my school, teachers provide individual attention 
to help the students learn better. 


Teachers at my school provide effective 
instruction. 


Teachers at my school assign meaningful 
homework on a regular basis. 


My school provides academically challenging 
curricula. 



0% 20% 40% 60% 80% 100% 


■ Negative ■ Positive 


Figure 20. Distribution of school personnel responses for the 
Support for Learning subscale. 

Note. Each item was measured on a 4-point Likert scale (1= Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly agree), 
ratings of I and 2 were grouped as a negative response and ratings of 3 and 4 were grouped as a positive response. 

Significant variations continue to exist between the i3 schools, F( 5, 486) = 2.90, 
p — 0.014, rf = 0.029. Yet, the results of multiple comparisons did not reveal any significant 
differences between individual schools. The results of subgroup analyses revealed that male school 
personnel reported a higher level of support for learning than did female school personnel, 

F(l, 444) = 15.53 ,p < 0.001, rf = 0.034. Additionally, school personnel from racial/ethnic minority 
backgrounds reported a lower level of support for learning than did their White counterparts, 

F(l, 424) = 14.23 ,p < 0.001, rf = 0.032. School personnel’s ratings on the Support for Learning 
subscale did not differ by years of teaching experience (i.e., school personnel with less than one year 
of teaching experience vs. school personnel with at least two years of teaching experience), 

F(l, 307) = 1.74, p = 0.188, rf = 0.006. 

Adult Social Support Scale. Five CSS items were selected to assess school personnel’s 
perceptions of adult social support for their students. Of the valid student responses ( n — 492), the 
scale mean was 3.08 ( SD — 0.61). As illustrated in Figure 21, school personnel reported feeling 
more positive than negative about adult social support items. Flowever, one item received a much 
lower rating as compared to all other items, “If I really try hard, I can get through to even the most 
unmotivated students” (72% positive responses); while one item receive 98% of positive responses, 
“ I feel the teachers at my school really care about their students.” 
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If teachers have willingness, they can help any 
student learn. 


18 . 6 % 


81 . 4 % 


By trying a different teaching method, I can 
significantly affect a student’s achievement. 


If I really try hard, I can get through to even the 
most unmotivated students. 


I believe students at my school can talk with their 
counselor or dean. 


I feel the teachers at my school really care about 
their students. 



0% 20% 40% 60% 80% 100% 


■ Negative ■ Positive 


Figure 21. Distribution of school personnel responses for the 
Adult Social Support subscale. 

Note. Each item was measured on a 4-point Likert scale (1= Strongly disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly agree), 
ratings of I and 2 were grouped as a negative response and ratings of 3 and 4 were grouped as a positive response. 

Significant variations continue to exist between the i3 schools, F( 5, 486) = 3.95, 
p — 0.002, yf — 0.039. School personnel from Waggener High School reported a higher level of 
adult social support than did school personnel from Moore Traditional High School (p = 0.031), 
Valley Traditional High School (p = 0.002), and Western High School (p = 0.024). The results of 
subgroup analyses revealed that school personnel ratings of adult social support did not differ by 
gender or minority status, F(l,444) = 0.71 ,p = 0.400, rp = 0.002 and jf’(l, 424) = 0.37, p — 0.546, 
rp = 0.001, respectively. However, school personnel who have been in the school for two to five 
years reported higher ratings of adult social support as compared to school personnel who were in 
the school for less than one year, F(l, 307) = 5.82 ,p = 0.016, rp = 0.019. 24 

Implementation Fidelity for PLCs 


The fidelity score for this project goal is high as most of the indicators have been mandated 
by the district. The indicators for this construct follow: 


• Frequency of content-based PLCs 

• Duration of content-based PLCs 


24 The coding for years of experience in this school was: I = less than one year, 2 = two to five years, 3 = six to 10 years, 
4 = I I to 20 years, and 5 = more than 20 years. 
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Frequency of cross-disciplinary PLCs 
Duration of cross-disciplinary PLCs 






Each project school was scored individually on these indicators. The scores were aggregated 
across the schools for one overall fidelity score. Table 45 shows the full fidelity table and scores. 

To be implemented with fidelity, at least 67% of the schools must obtain a minimum score of 10. 
During Year 5, none of the schools scored a 10; therefore, this component has not implemented 
with fidelity. 
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Table 45. JCPS Fidelity Measure and Scores for Component 3: PLCs 


Key Elements of 
Component 

Operational 
Definition 
for Indicator 

Data 

Source(s) for 
Measuring the 
Indicator 

Data 

Collection 
Schedule for 
the Indicator 

Raw Scoring and Scoring Criteria for Fidelity 

Possible 

Indicator 

Score Per Scoring School Indicator 

School Criteria Name Score 

1 

Content-based 

PLCs 

Frequency of PLCs 

Teacher surveys 

Teacher survey 

administered 

annually 

1-4 

/ = Low 

Fewer than 1 0 content-based 
PLCs annually 

2 = Minimal 

Academy @ 
Shawnee 

4 






Fern Creek 

4 







11-12 content-based PLCs 
annually 

Moore 

4 







3 = Adequate 
13-14 content-based PLCs 
annually 

4 = High 

Valley 

4 







Waggener 

4 







15 or more content-based PLCs 
annually 

Western 

4 

2 

Content-based 

PLCs 

Duration of PLCs 

Teacher surveys 

Teacher survey 
administered 

1-4 

1 = Low 

Less than 50 hours 

Academy @ 
Shawnee 

1 





annually 


2 = Minimal 

Fern Creek 

1 







50-54 hours 

3 = Moderate 
55-59 hours 

Moore 

1 







Valley 

1 







4 = High 
60 or more hours 

Waggener 

1 







Western 

1 

~1~ 

Cross-disciplinary 

PLCs 

Frequency of 
cross-disciplinary 

Teacher surveys 

Teacher survey 
administered 

1-4 

1= Low 

Fewer than 25 cross- 

Academy @ 
Shawnee 

1 



PLCs 


annually 


disciplinary PLCs annually 

Fern Creek 

1 
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Raw Scoring and Scoring Criteria for Fidelity 


Key Elements of 
Component 

Operational 
Definition 
for Indicator 

Data 

Source(s) for 
Measuring the 
Indicator 

Data 

Collection 
Schedule for 
the Indicator 

Possible 
Indicator 
Score Per 
School 

Scoring 

Criteria 

School 

Name 

Indicator 

Score 







/ - Minimal 

25-28 cross-disciplinary PLCs 

Moore 

1 







annually 
2 = Adequate 

Valley 

1 







29-31 cross-disciplinary PLCs 
annually 

Waggener 

1 







3 = High 

32 cross-disciplinary PLCs 
annually 

Western 

1 

4 

Cross-disciplinary 

PLCs 

Duration of cross- 
disciplinary PLCs 

Teacher surveys 

Teacher survey 
administered 

1-4 

1 = Low 

Less than 22 hours 

Academy @ 
Shawnee 

1 




annually 


2 - Minimal 

Fern Creek 

1 







22-24 hours 

3 = Moderate 
25-28 hours 

Moore 

1 







Valley 

1 







4 = High 
29 or more hours 

Waggener 

1 







Western 

1 




10-14 = 

Academy @ 
Shawnee 

7 




High Implementation 

Fern Creek 

7 

Coirmonent Level Score 

1-16 

6-9 = 

Moore 

7 




Moderate Implementation 

Valley 

7 




1-5 = 

Waggener 

7 




Low Implementation 

Western 

7 
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Raw Scoring and Scoring Criteria for Fidelity 




Data 

Data 

Possible 





Operational 

Source(s) for 

Collection 

Indicator 




Key Elements of 

Definition 

Measuring the 

Schedule for 

Score Per 

Scoring 

School 

Indicator 

Component 

for Indicator 

Indicator 

the Indicator 

School 

Criteria 

Name 

Score 


Implementation with fidelity = At least 67% of the schools have a minimum score of 10 or above 
Implementation without fidelity = Fewer than 67% of the schools have a score of 10 or above 


No school has a score of 
more than 1 0; thus, this 
component has not been 
implemented with fidelity 
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Summary and Recommendations 


Project Goal I : Student Mastery of Academic Material 

Project Goal 1 included objectives regarding course assignment protocols and monitoring of 
progress; core course analysis; and perceptions of academic challenge from both the student and 
teacher perspectives. Progress on this goal was analyzed through a variety of measures. Evaluators 
interviewed school counselors to learn how they assign students to courses and monitor students’ 
progress to ensure they are in the appropriate classes. Teachers were also asked on the Teacher 
Survey to respond to questions about the master schedule, while principals were asked questions on 
the implementation survey regarding students’ course placement and scheduling. In addition, 
teachers addressed questions regarding the level of academic challenge provided to students at their 
school. Students’ perceptions of academic challenge, teacher support for learning, and self-efficacy 
were also measured through the CSS and Student Survey. To measure students’ academic progress, 
core course pass and fail rates were analyzed. 

Course Assignment Protocols and Monitoring of Progress 

Although the trimester was the driving force behind this goal, due to external forces, 
including the implementation of end-of-course exams, the trimester schedule did not work how it 
was intended. Initially, the trimester was designed to provide students, especially those who were 
struggling, with multiple opportunities to pass courses so they could graduate on time. With the 
implementation of end-of-course exams, however, many core content courses (i.e., English and 
math) became three-trimester or year-long courses. 

According to the interviewed counselors, the trimester allowed for scheduling flexibility and 
the ability to immediately respond to student needs. Simultaneously, it was challenging because they 
had to change the master schedule each trimester. One of the counselors also believed that by 
providing the students with multiple opportunities to pass a course within the trimester schedule, 
students were given an open invitation to fail a course without major consequence because they 
were able to take it many times. For accelerated students, one counselor indicated that the trimester 
was “irrelevant” because most accelerated students took year-long classes while another counselor 
thought that the trimester limited opportunities for advanced students because they could not take 
as many electives or participate in the career pathway curricula. Based on Teacher Survey results, 
the majority of teachers did not think that the trimester schedule met the needs of either struggling 
or advanced students. 

In terms of placing students in courses, the counselors discussed how they use multiple 
sources of data to accurately assign the student to the course that best fits his or her needs (e.g., 
comprehensive, advanced, honors, or AP). For upperclassmen, this typically involved conversations 
with the students to discuss potential interests to help frame a course load around those interests in 
addition to meeting graduation requirements. For incoming freshmen, the counselors also review 
data from the students’ middle school career. At least one counselor from the high school typically 
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meets with students at the middle schools to discuss high school graduation requirements and 
expectations. 

In terms of students’ perceptions of the scheduling process, over half of the Student Survey 
respondents (56%) agreed that they were able to change courses if they were struggling in a class, 
and 82% agreed that there are opportunities to make up courses in which they have failed. Students 
were also asked if they had opportunities to take courses outside their core courses and if they had 
opportunities to take honors or advanced classes. Over 80% of the respondents agreed with both of 
those statements. 

Recommendations 

• Request that counselors and other affiliated school personnel document their scheduling 
and monitoring processes. From these submissions, create a “Best Practices” resource 
that will provide a multitude of ideas that schools can adapt to fit into their own 
contexts. 

• Bring schools together to share their successes and challenges. During this time, have 
district staff present information on who can address questions that master schedulers 
and counselors may have that relate to scheduling (e.g., Infinite Campus) or student 
placement. 

• Develop case studies of specific programs (e.g.. Schools of Study) which may warrant 
further study and be of interest to the schools throughout the district. 

• Have the counselors facilitate meetings with the faculty at their schools to learn more 
about why teachers indicated on the teacher survey that the schools may not always have 
the best placement strategies for students. 

• Create time for school staff to discuss how the current scheduling process impacts both 
struggling and accelerated students as they were divided on whether the current 
processes benefitted these students. Listening to different perspectives may encourage 
and create new ideas that can benefit all students, regardless of their academic status. 

Core Course Pass Rates 

Overall, approximately 82% of i3 students passed their core courses. Across the subject 
areas, the pass rates were higher in English (86%) and social studies (84%) than in math (79%) and 
science (77%). As compared to the previous year, i3 students made some progress in English 
(1% increase) and social studies (1% increase), but they showed a decrease in math (1% decrease) 
and science (3% decrease). Thus, the i3 schools did not meet any of the Year 5 performance targets 
across any subject area (6% to 1 1% below the targets). 

Although student pass rates varied across different subjects and core courses, one school 
(i.e., Moore Traditional High School), in particular, had higher pass rates than did others. While the 
majority of schools struggled to meet the Year 5 targets, Moore Traditional High School exceeded 
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the Year 5 target (1% above the target) in social studies and was only 0.6% shy from the science 
target. 


Findings of subgroup analyses revealed that trimester scheduling seemed to have positive 
effects in English, math, social studies, and some sub-areas of science (i.e., life science and 
earth/ space science) in comparison with the traditional two-term courses. Additionally, 
achievement gaps measured by course pass rates continued to be observed among the subgroups. 

In particular, female students had higher passing rates as compared to male students across all 
subject areas and across the majority of the core courses. Nevertheless, consistent with national 
trends, students from certain racial/ethnic (i.e., Black/ African American) and socioeconomically 
disadvantaged (i.e., those with FRPM status) groups fell behind their counterparts. Such differences 
were more pronounced between White students and Black students and between students from 
middle- and upper-income families (i.e., students with paid meal status) and low-income families 
(e.g., students with FRPM status). These gaps have been well documented in the literature as well as 
in the previous project reports. Yet, it is interesting to observe that, in Year 5, Hispanic students 
demonstrated higher pass rates in math, science, social studies, and the majority of the core course 
areas than they have in past years. 

Recommendations 

• Identify, develop, and implement strategies to close achievement gaps among the 
subgroups, specifically the ethnicity/ race and FRPM subgroups. 

• Examine why specific courses (e.g., Geometry, Physical Science, and Earth/Space 
Science) tend to have lower pass rates than other core courses. Determine which 
strategies teachers may need to utilize to help increase pass rates. Provide professional 
development for teachers to learn how to effectively implement these instructional 
strategies in their teaching. 

• Study the variations of the pass/ fail rates across the schools to determine why some 
schools seem to have higher pass rates than others. Use data from this report and/ or 
school data to create individual school profiles to look at trends, ask more in-depth 
questions, and gain a full understanding of what is happening at each school, specifically 
in terms of the achievement gaps. Develop an action plan based on these findings to 
support the school in closing the achievement gaps. 

Academic Challenge, Self-Efficacy, and Support for Learning from Teachers 

Overall, the i3 schools showed a slight increase (2%) in the mean percentage of students’ 
perceptions of academic challenge from Year 4 to Year 5. When compared to the end-of-project 
target, i3 schools, on average, were 8% below the goal of 75% agreement. The examination of 
variations across the i3 schools revealed that four schools (i.e., Fern Creek Traditional High School, 
Moore Traditional High School, Valley High School, and Waggener High School) showed minor 
increases from Year 4 to Year 5; yet, the other two schools (i.e.. Academy @ Shawnee and Western 
High School) showed minor decreases from Year 4 to Year 5. Further, results of subgroup analyses 
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revealed that students’ perceptions of academic challenge differed by their LEP status, grade level, 
and race/ ethnicity. 

Teachers tended to agree that the school provided students with academic challenge. A 
majority of teachers agreed that the students seemed to enjoy school and that high grades are viewed 
as an indicator of mastering high standards. However, a third also agreed that students were bored 
at the school and just over half agreed that students do well at their school. 

Responses regarding student engagement were also positive. Teachers agreed that they 
challenge their students academically and help disenfranchised students feel more connected to 
school. They also tended to agree that the teachers in their school encourage students to go beyond 
stated expectations and involve the students in setting those expectations. However, a majority of 
teachers disagreed that they involve students in developing criteria for assessing their assignments 
and that they work to involve students in planning lessons. Thus, it appears that teachers still 
struggle with ways of encouraging student voice in the classroom. 

Recommendations 

• Investigate variations across the schools to identify strategies and practices that are 
working in some schools but not working in other schools. Experiment with modifying 
strategies and practices for the respective schools. 

• Conduct focus groups with small groups of students across the grade levels to learn 
more about what a “challenging” curriculum means to them, and what they believe 
constitutes “meaningful” homework assignments. 

• Facilitate conversations with the leadership at each of the i3 schools to review the data 
regarding students’ perceptions of academic challenge, which differed across the schools 
and may suggest that some schools need more support than others. Develop an action 
plan with each school to improve students’ perceptions of academic challenge. 

• Encourage building leadership to have conversations with teachers about how to 
develop an engaging and challenging curriculum. If possible, allow students to join in 
those conversations in order to have their voices heard and to establish a new type of 
relationship with teachers and administrators. 

Fidelity 

Each school has implemented Project Goal 1 related to student scheduling with a high 
degree of fidelity. The elements assessed by the fidelity tool included: (1) master scheduler assigned; 
(2) five 70-minute courses per day; and (3) students assigned to courses based on data (e.g., academic 
history and assessment scores). Because at least 67% of the project schools received a score of 5 or 
higher, this component has been implemented with a high level of fidelity. 
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Project Goal 2: Student Engagement in School and College Readiness 


The second project goal is to provide a range of personalized supports to students to 
increase engagement in school and promote college readiness. The primary strategy is to create 
CAT advisory periods for students that are designed to focus on college readiness and support 
students’ sense of community within their schools. In addition to CAT implementation data 
(gathered via the CART and principal surveys), 10th grade PLAN scores and 11th grade ACT scores 
were analyzed to measure students’ college preparedness. Postsecondary transition rates were 
examined through StudentTracker data. Students’ perceptions about their own engagement, adult 
social support, and peer social support were measured through the CSS while their perceptions of 
CAT were measured by the Student Survey. 

CAT Implementation 

To support college readiness for JCPS students, all project schools hired a CART to design 
and plan the CAT advisory periods to increase students’ understanding and interest in attending 
college. Being that the CARTs have the autonomy to plan and design CAT advisory periods within 
each project school, they have implemented projects and have curricula planned out for the school 
year. However, they still struggle with having 100% of the teachers buy into the process and engage 
students. Further, a few of the CARTs mentioned that student apathy is an issue. The CARTs also 
had limited time and opportunity to provide training to their own staff, and there were limited 
opportunities for them to receive training for themselves. 

Recommendations 

• Allow time for the CARTs at the i3 schools to visit other i3 schools (or other district 
schools) to monitor, assess, and provide feedback on how to strengthen each other’s 
programs. This would allow for the creation of a community of learners to be built 
within the i3 project schools, which could be expanded to other CAT programs in the 
district. 

• Provide training at the district level for all advisors so those assigned as advisors can 
continue to see the value of the program and increase their buy-in. 

• Encourage building administrators to allow time for the CARTs to provide training to 
advisors at their schools. This may also help administrators who have not fully bought 
into the program to become more aware of its value and to help build support for CAT. 

• Create more structured guidelines for what a CAT advisory period should look like. 
Although the CART s appreciate the autonomy afforded to them, structure and CAT 
standards would help them reach targeted goals. 

• Share information with the CARTs on funding resources, particularly for schools where 
the CART position may not be full-time in the future. 
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Students * Knowledge and Use of College Readiness Skills and Postsecondary 
Transition 

Based on students’ ACT and PLAN outcomes, a finding from this evaluation was that the 
majority of i3 students are not college ready, using ACT and PLAN benchmarks as an indicator of 
college readiness. Compared to Year 4 findings, the performance of the i3 schools declined in all 
subject areas in Year 5. Although the schools still have not reached the project targets, the gaps 
were reduced across all subject areas. Gaps between performance and the end-of project goals range 
from 13 to 19 percentage points. 

The percentage of i3 students meeting or exceeding the ACT benchmarks decreased in 
English and reading from Year 4 to Year 5 but increased in math. Overall, the gap to the end-of- 
project target ranges from 6% to 14%, with the largest gap in reading, followed by science, math, 
and English. 

In regards to postsecondary transition, there was an overall increase in the percentage of 
students enrolling in a postsecondary institution from 2013 (38%) to 2014 (45%). Although the 
target goal of 55% was not reached, there was an increase of 4% from the baseline to 2014. It 
should also be noted that of those students who did enroll in postsecondary institutions, 

86% attended public institutions, 53% attended four-year institutions, and 90% attended an 
institution in Kentucky. 

Recommendations 

• Review instructional practices to determine why student scores are still not reaching the 
PLAN and ACT benchmarks, specifically in the areas of math and science. Provide 
professional development to teachers who may benefit from new pedagogical 
instruction. 

• Establish a time for i3 school administrators, teachers, and CARTs to share their lessons 
learned or best practices of what has or has not worked in their schools to improve 
academic achievement. Some schools have made more progress than others. Start with 
a team at those schools to learn more about what they have done to achieve their current 
accomplishments. 

• Strategize to further reduce achievement gaps among subgroups, particularly 
racial/ethnic groups and students qualifying for FRPMs. 

• Continue to examine postsecondary transition data and address questions regarding why 
student transition rates have decreased. 

• Encourage the CARTs to conduct exit interviews with seniors to find out why they made 
particular choices. CAT should be a time for students to set goals and work with their 
advisor to ensure that they achieve those goals. Track the progress of students; if 
students have developed relationships with their advisors over the course of the school 
year, encourage both the advisor and student to send an e-mail in late summer or early 
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fall to track what the student is doing and why that student chose the path that he or she 

did. 

Student Perceptions of School Climate 

Multiple CSS items were used to examine students’ perceptions of (1) support for learning 
(1 item), (2) adult social support (5 items), (3) peer social support (1 item), and (4) school 
connectedness (3 items). Overall, students reported more positive than negative perceptions across 
all constructs. Two items were also selected from the CSS to assess student engagement. About 
one third of students indicated that they were engaged in sports or clubs at school 

Student perceptions also varied by school. Overall, Fern Creek Traditional High School 
reported higher levels of support for learning, adult social support, peer social support, and school 
connectedness than did all other i3 schools. Student perceptions also differed by students’ 
demographic characteristics. As seen in Table 45, male students reported higher levels of peer social 
support and school connectedness than did female students; yet, female students reported a higher 
level of adult social support than did male students. Students from higher grade levels reported 
higher levels of support for learning, adult social support, and school connectedness. Asian 
students, in general, reported higher levels of adult social support, peer social support, and school 
connectedness as did students from other racial/ ethnic groups. Student perceptions of support for 
learning, peer social support, and school connectedness did not differ by FPRM status; yet, students 
with paid meal status reported a higher level of adult social support than did students with free meal 
status. Students with LEP also reported higher levels of support for learning, adult social support, 
peer social support, and school connectedness than did students without LEP. 


Table 46. Snapshot of Overall Student Perceptions of School Climate 


Demographic 

Characteristics 

Support for 
Learning 

Adult Social 
Support 

Peer Social 
Support 

School 

Connectedness 

Gender 

Not significant 

Female > Male 

Male > Female 

Male > Female 

Grade level 

1 2th > 9th 

1 Ith & 12th > 
9th & 10th 

9th > 10th 

1 Ith > 10th 

Race 

Hispanic > White 

Asian > Black 

Asian > White, 
Black, Hispanic 

Asian > Hispanic > 
White & Black 

FRPM 

Not significant 

Students with paid 
meal status > 
students with free 

Not significant 

Not significant 



meal status 



LEP 

LEP > non-LEP 

LEP > non-LEP 

LEP > non-LEP 

LEP > non-LEP 


Recommendations 

• Review the school culture data with the i3 schools. Develop action plans on how the 
staff at each school can continue to improve their climate. 

• Continue to develop a culture within the schools that provides peer-to-peer support to 
ensure there is a respectful culture between students as well as adults. 
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• Work with building administrators to develop stronger relationships with families and 
community members. Continue to emphasize afterschool programs and activities while 
ensuring that all students have access. 

• Investigate why grade levels differed across the various CSS constructs. For example, 
why do 1 2th graders believe there is more support for learning than ninth graders? If 
there are certain activities happening at a particular grade level to make them feel more 
supported, how can those activities be replicated in other grade levels? 

• Explore why students with LEP tend to have more positive perceptions of school 
culture. If it is because of special services they receive, brainstorm ways in which other 
students can receive more personalized services to meet their individual needs. 

Fidelity 

No schools on this component reached a threshold of implementation. However, it should 
be noted that the indicators are based upon JCPS’ original proposal on what they thought was 
achievable. Once the i3 schools began to implement the CAT advisory periods, they found that 
adjustments had to be made to fit CAT within the school schedule. For example, not all of the 
schools could implement a 55-minute CAT period due to scheduling conflicts. Other schools 
decided that they only wanted to offer CAT biweekly in order to increase academic time. Thus, the 
goals originally set for CAT in the proposal were not feasible for all participating i3 project schools. 
During Year 5, response rates on the survey and interviews were also low, so some of the fidelity 
measures were based on informal conversations with the CARTs and past trends. 

Recommendations 

• Discuss the fidelity components with school leaders and teachers so they have a clear 
understanding of the expectations of CAT (e.g., how many times per week the advisory 
periods should be held and how many minutes they should last). 

• Provide relevant, useful, and timely professional development sessions for CARTs, 
especially on ways in which they can provide training to advisors in their schools. Focus 
on how they can train advisors to engage and motivate students. 

• Develop a common walkthrough tool and provide guidance to the CARTs on how to 
provide feedback to the advisors after they have been observed by the CART. Establish 
a continuous improvement model for CARTs to use with their advisors in order to 
strengthen the relationships between advisors and students. 
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Project Goal 3: Teachers’ Pedagogical and Student Support Practices 


The third project goal is to improve teachers’ pedagogical and student support practices to 
maximize the effectiveness of increased learning time. The strategy to reach this goal is to ensure 
that content-based and cross-disciplinary PLCs meet regularly over the school year. The increase in 
the number of PLCs should lead to improvements in teachers’ perceptions of collaboration and self- 
efficacy. In terms of instructional practice, teachers’ perceptions of how their own instructional 
practices increase student academic engagement and challenge should improve, as well as their 
perceived ability to identify and appropriately respond to students’ unique academic and social 
needs. Evaluators collected relevant data via the CSS and Teacher Survey to assess this project goal. 

Professional Learning Community (PLC) Meetings 

Findings showed that more time is allocated to content-based PLCs than cross-disciplinary 
PLCs. Based on survey responses, on average, the content-based PLCs met weekly across the 
project schools while the cross-disciplinary PLCs met less frequently. Survey results showed that 
half of the i3 project schools hosted cross-disciplinary PLCs monthly while the other three schools 
met once a trimester. Principals reported that they were pleased with the implementation of both 
types of PLCs at their schools. Typical challenges that were shared included lack of time and 
monetary resources to continue to support the PLCs. 

Recommendations 

• Provide continued assistance to schools to help them find funding for the provision of 
stipends to teachers for participating in activities outside of their normal duties or to pay 
for substitutes so teachers can participate in “data days”. 

• Allow principals and teacher leaders to visit other high schools to observe their PLCs 
and other effective collaboration practices. 

• Communicate with principals and teachers to let them know how the PLCs will be 
continued without the support of the i3 grant as concerns were raised about this issue. 

• Review the target benchmarks to determine if modifications need to be made to better 
reflect how the PLCs operate in practice. 

• Establish districtwide guidelines on how often content-based and cross-disciplinary 
PLCs should meet. Ensure that each school understands how JCPS defines a PLC so 
school personnel consistently record meetings. 

• Systemically collect information about the how schools facilitate PLCs. Create a “Best 
Practices” document that describes effective practices across schools. Work with the 
evaluation team to create a data collection system for the project schools that could be 
adapted across the district. Facilitate a meeting with the schools to share “Best 
Practices” and help them learn from one another. 
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Increase Teachers * Perceptions of Collaboration 

Teachers tended to have higher levels of agreement on items regarding teachers supporting 
one another, collaborating, and working together. This may be a result of the PLCs, where time is 
devoted for teachers to work collaboratively to discuss students’ progress and challenges in a 
supportive environment. Teachers were less likely to agree on items regarding resources and time. 

A large percentage of teachers disagreed with the statements about having enough time to 
collaborate with colleagues to improve instruction and the ability to access expertise and sufficient 
resources in a timely fashion. Another survey item that did not have a high level of agreement was 
“Teachers in this school trust each other.” However, it takes time to develop trust and a truly 
collaborative culture, and it seems that the schools are continuing their work in building a tmsting 
culture within their schools. 

Recommendations 

• Continue to provide professional collaboration opportunities for teachers with the 
inclusion of trust exercises. Although collaboration is building within the schools, results 
for this area indicate that tmst issues remain within the schools. 

• Work with building administrators to ensure that the schools have adequate time and 
resources for collaborative planning and PLCs. The district can provide ideas and 
strategies to help the schools meet the scheduling needs of both students and teachers. 

• Survey teachers and building administrators to learn more about what resources may be 
lacking or what resources they believe will help them improve their teaching practices. 

• Communicate with the schools about how collaboration will continue to be supported 
by the district once the i3 funding is gone. 

Improve Teachers * Perceptions of Their Own Self-Efficacy with Respect to 
Content-Based Knowledge for Teaching 

In general, teachers tended to have high levels of self-efficacy as over 80% of them 
responded positively. The items that did not score as highly were about pressure to achieve 
immediate results. Only a quarter of the teachers responded that they never or seldom worry about 
being criticized if positive results are not readily available and over half (59%) are sometimes or almost 
always discouraged from trying new approaches because of the emphasis on success. 

Recommendations 

• Provide training sessions for school staff to help them look more positively at data so they 
can measure progress by the “smaller” successes and look at trends over time rather than 
immediate results. 

• Document innovative teaching strategies being used with success. Build time into schedules 
to allow teachers to share innovative practices that have led to student success. 
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• Work with administrators on ways to motivate staff and develop a culture that does not 

include fear of being criticized and one that promotes innovative techniques, as long as 

results are being measured and continuously improved. 

Improve Teachers * Perceptions of Their Instructional Practices in Order to 
Increase Student Academic Engagement and Challenge 

Teachers tended to rate survey items that focused on challenging students and working with 
disenfranchised students higher than items related to students being actively involved in planning 
lessons, setting expectations, and developing criteria for lessons. It appears that teachers have fewer 
tendencies to actively involve students in planning lessons, although teachers perceived themselves 
to be challenging students academically and setting high expectations for them. 

Recommendations 

• Continue to work with staff to encourage students to go beyond the expectations set by 
teachers. Provide training to help motivate students who may have apathetic tendencies. 

• Encourage classroom teachers to collaborate with the students’ advisors to encourage 
and motivate them to go beyond classroom expectations. 

• Assist teachers in developing lesson plans that will encourage an increase in students’ 
involvement (i.e., student voice) within the classrooms. 

• Develop protocols that teachers can use in the classroom as they work to increase 
student voice. Because this may be a different teaching model than what teachers are 
typically used to, the district may need to expend time and resources to gain buy-in from 
teachers in understanding how this type of teaching strategy may improve student 
outcomes. 

• Conduct focus groups with a small sample of students to collect their input on how best 
to increase their voice and active engagement in terms of lesson development and 
planning. Have a pilot group of teachers implement those strategies to document what 
works and share their successes and challenges with other teachers in their building and 
throughout the district. 

Improve Teachers * Perceptions of Their Ability to Identify and Appropriately 
Respond to Students * Unique Academic and Social Needs 

Overall, school personnel reported feeling more positive than negative about the adult social 
support and support for learning items. However, one item, in particular, received a much lower 
rating as compared to all other items, “Teachers at my school assign meaningful homework on a 
regular basis” (55% positive responses). Additionally, school personnel’s ratings of adult social 
support and support for learning differed by school. School personnel ratings also differed by the 
demographic characteristics of the staff; however, the patterns are inconsistent across the two 
different subscales. Specifically, in terms of support for learning, male and White staff members 
reported higher ratings than did female staff and staff from racial/ ethnic minority backgrounds. 
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Regarding adult social support, staff with more than two years of teaching experience perceived a 
higher rating than did staff with less than one year of teaching experience. 

Recommendations 

• Observe teachers using the district observation protocol to determine if they are 
effectively differentiating instruction for all students. 

• Provide professional development on differentiated instruction and how to personalize 
instruction for individual students to teachers who may be struggling in this area. 

• Conduct focus groups or interviews with students to learn more about how they think 
teachers could provide individualized support for them. 

• Encourage dialogue between classroom teachers and students’ assigned advisors to 
further explore how instmction can be individualized for each student. 

PLC Implementation Fidelity 

No schools on this component reached a threshold of implementation. However, the 
indicators are based upon the original proposal. Project staff may want to reconsider whether or not 
these are appropriate targets given how the schools ended up structuring their PLC sessions. 

Recommendations 

• Create a feedback mechanism to monitor the extent to which teachers implement 
instructional strategies discussed within the PLCs. This may be accomplished through 
the use of an administrator observation form or a district feedback form to gather more 
data about how the work of the PLCs are being implemented in the classroom. 

• Continue to support cross-disciplinary PLCs by providing staff with time to participate 
and resources. Document how the cross-disciplinary PLCs are functioning by 
conducting focus groups or interviews with leaders of those PLCs. Share the findings 
with the i3 schools that may be struggling to implement this component. 
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Principal Interview Protocol - Spring 20 1 5 


Investing in Innovation (i3) Implementation Evaluation 


Overview 

1. Please tell me what, if any, have been the biggest changes at [high school name] over the 
course of the 2014-2015 school year. 

2. In general, what are your goals for the school and how do you perceive the i3 grant helping 
you to achieve those goals? 

Professional Learning Communities (PLCs): Course-Based 

1. What, if any, changes have been made to the structure of the PLCs over the course of the 
2014-2015 school year? 

2. If I were to observe a typical course-based PLC, what would I say? Is this in any way 
different from how such a PLC would have been structured or run in prior years? 

3. Who monitors the progress of the PLCs? How? 

4. Have you observed any benefits arising from the use of PLCs? 

Probe: Have you noticed any improvement in teachers’ instructional practices that you think are attributable 
to PLC participation ? 

5. What factors have supported the implementation of the course-based PLCs? 

Probe: How has the PLC Design Team provided support for implementation ? 

6. What have been the challenges associated with implementing course-based PLCs? How 
have you addressed those challenges? 

7. What types of professional development have you received to help in implementing a PLC? 

Probe: Who provided the professional development ? What types of resources and support did you receive ? 
Were the resources and support adequate? 

8. What other support or resources do you need? 

Professional Learning Communities: Cross-Content (School of Study Design Team) 

1. What, if any, changes occurred to the School of Study Design Team (SOS) PLC at [High 
School] for the 2014-2015 school year? 

Probe: When did the program get started? How many minutes do they meet each week? How did you decide 
who would be on the team (eg., volunteers, selection process)? 

2. If I were to observe an SOS PLC, what would I see that might be different from a non-SOS 
PLC? 
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3. Who monitors the progress of the SOS PLC? How? 

4. What benefits have you observed from the SOS PLC? 

Probe: Have you noticed any improvements in teachers’ instructional practices that yon think are attributable 
to SOS participation ? 

5. What factors have facilitated the implementation of the SOS PLC? 

6. What have been the challenges associated with implementing the SOS PLC? How have you 
addressed those challenges? 

7. What professional development have you received to help in implementing the SOS? 

8. What other support or resources do you need? 

Trimester Scheduling and Responding to Student Needs 

1. Has trimester scheduling met the needs of struggling students? Why or why not? How has 
the trimester schedule offered expanded learning opportunities for struggling students? 

2. Has trimester scheduling met the needs of high-performing students? Why or why not? 

How has the trimester schedule offered acceleration and expanded learning opportunities for 
higher performing students? 

3. How is your staff trained to monitor student progress? Do you have a process for a student 
to change courses if there is a placement more appropriate for him or her (i.e., a struggling 
student needs to be moved to a less challenging course while an advanced student need to be 
moved to a more challenging course)? 

4. Has the change of the accountability model (e.g., end-of-course exams) altered your thoughts 
and support of the trimester scheduling model? Why or why not? 

College Access Time 

1. What goals did you have for CAT this year? Did the CATs accomplish those goals? Why or 
why not? If they did not, what changes will you make to achieve those goals next year? 

2. What feedback have you received about CAT from: 

a. Teachers 

b. Students 

c. Parents 

3. What factors facilitated the implementation of CAT at your school? 

4. What have been the challenges associated with implementing CAT at your school? 

5. What additional support do you need to implement CAT at your school? 
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Sustainability 


1 . What piece or pieces of the grant do you think [high school] will be able to support once 
grant funding ends? 

2. What have been the two or three most important lessons learned over the course of the 
i3 grant? 
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College Access Resource Teacher Interview Protocol - Spring 2015 


Investing in Innovation (i3) Implementation Evaluation 

College Access Time (CAT) Implementation 

1. Briefly describe your current responsibilities and discuss any changes that have occurred in 
your role as CART over the course of the year. 

2. Please describe what a typical CAT advisory looks like in your school now. 

Probe: Do you have different curricula for different grade levels ? How do you focus on building a strong 
relationship with an adult who is “on my case and on my side” and positive peer relationships around 
common career interests, and college readiness ? 

3. What were your goals for CAT this year? Have you achieved these goals? How? 

Probe on the following four goals of CAT: 

a. Advance college -ready skills, monitor student progress, and increase students’ 
sense of affiliation with adults and peers 

b. Build knowledge of and use of college -ready skills and habits 

c. Improve students’ sense of affiliation with adults and peers 

d. Strengthen peer-to-peer relations and perceptions of peer support, particularly 
within career themes 

4. Tell me how your design team has functioned during the 2014-2015 school year. What types 
of topics and tasks did it address? How often did it meet? 

5. How have you monitored CAT implementation at your school? What, if any, changes have 
you made to CAT based on your monitoring processes? 

6. What factors have supported CAT implementation? 

7. What have been the greatest challenges associated with CAT implementation? 

Advisor Training 

1 . How have you provided training to advisors throughout the school year? 

2. How have teachers responded? What challenges have you faced? How did you address 
those challenges? 

CAT Logistics 

1 . Our records show that your CAT is offered [INSERT NUMBER OF MINUTES CAT IS 
OFFERED FROM SPRING 2015 SURVEY], Did you make any changes to the number of 
minutes CAT is offered each week? 
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2. Did you make any changes to the student-to-advisor ratio over the course of the school year, 
which was [INSERT STUDENT TO ADVISOR RATIO FROM SPRING 2015 
SURVEY]? 

3. Did you collect data from students, and if yes, how (e.g., surveys, focus groups)? How did 
student feedback impact CAT implementation? 

CART Training 

1. What professional development resources did you receive from JCPS (e.g., training or 
materials) for your own growth? Were the resources provided adequate? Was adequate 
support provided for using the resources? 

2. What other support or resources do you need from the district? From your school? 

Sustainability 

1 . How will advisory continue at [high school] once the funding ends? 

2. What have been the two or three most important lessons learned about advisory over the 
course of the i3 grant? 
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Counselor Interview Protocol: 

Course Assignment and Response to Student Needs 


Investing in Innovation (i3) Implementation Evaluation 


Objectives: 

(1) To determine how school counselors assign students to courses, including how freshmen 
placement differs from upperclassmen placement. 

(2) To determine if the trimester schedule meets the needs of the students in terms of course 
placement (e.g., appropriate remediation and acceleration). 

(3) To determine whether, and if so, how, student progress (for the purpose of class placement) 
is monitored throughout the year. 


1. Walk me through the process you used to place rising 10th-, 1 1th-, and 12th-grade students 
in classes during the 2014-2015 school year. 

a. When did you begin the course placement process? 

b. Who, besides yourself, was involved in the course placement process? 

c. How did you determine student placement (e.g., data, parent input, or other)? 

d. How did you and under what circumstances did you make changes to individual student 
schedules during the 2014-2015 school year (i.e., how did you monitor progress)? 

e. Did you receive any sources of support during the student placement process? What 
kind of support did you receive and from whom (e.g., district)? 

£ Have you participated in any professional development or training related to the 
scheduling of students? 

2. Did you use the same process for placing ninth graders? If no, please explain how you made 
freshmen placements. 

3. What, if any, changes did you make in placing students from the 2014-2015 school year to 
the 2013-2014 school year? 

4. Are there other changes that need to be made to the course placement process? 

5. How well does trimester scheduling meet the needs of struggling students? Why or why 
not? How has the trimester schedule offered expanded learning opportunities for stmggling 
students? 

6. How well does trimester scheduling meet the needs of high-performing students? Why or 
why not? How has the trimester schedule offered acceleration and expanded learning 
opportunities for higher performing students? 
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7. In what ways has the trimester schedule affected student achievement? Why do you think 
the trimester schedule has affected student achievement in that way? 

8. In what ways has the trimester schedule affected graduation rates? Why do you think the 
trimester schedule has affected graduation rates in that way? 

Sustainability 

1 . Will you school maintain trimesters? Why or why not? 

What have been the biggest lessons learned about scheduling students over the course of the 
i3 grant? 
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Spring 2015 Principal Implementation Survey 


Jefferson County Public Schools (JCPS) 

Investing in Innovation (i3) 

Principal Survey 
Spring 20 1 5 

McREL International (formerly the Mid-Continent Regional Education Laboratory and Edvantia, 
Inc.), an education research company, is working with the Jefferson County Public Schools (JCPS) 
district to evaluate the Investing in Innovation (i3) project. Project staff want to know about your 
thoughts, experiences, and so on related to your involvement in the implementation of the i3 project 
so they can use that information to make adjustments and improvements. Data from this survey 
will be summarized in a report provided to JCPS project staff. The survey should take 
approximately 25 to 30 minutes to complete. Please complete by Friday, May 1. 

Protecting Your Rights: Participation in this survey is voluntary and should not involve any 
known risks above those normally encountered in daily life. There will be no retribution of any sort 
for the responses you provide. If you feel uncomfortable about responding to some questions, 
please feel free to skip those questions. You may also choose to stop participating in the survey 
altogether without penalty or reprisal. 

About Your Confidentiality: Survey responses will be collected via Qual tries utilizing transport 
layer security (TLS) encryption. TLS encryption is a security feature that encrypts the survey link 
and data as it moves across the Internet. McREL evaluators will handle the data and do everything 
they can to ensure its security 25 . Although most responses will be aggregated across schools, please 
note that your responses may be identified by school. This will help us to identify practices being 
implemented at schools that may warrant further study. 

Benefits: Although there are no direct personal benefits for you and you will not receive any 
compensation for participating in the survey, the information you provide will give project staff 
better information about how to improve the project. These kinds of improvements may result in 
indirect benefits to you as a participant in the project. 

Questions: If you have any questions about this survey, please contact Dr. Tara Donahue, 
Managing Evaluator (800.624.9120 ext. 5419; tdonahue@mcrel.org). For information on protection 
of your rights as a participant, contact Karen Bumgardner, a member of McREL’ s Institutional 
Review Board, at 800.624.9120, ext. 5841, or kbumgardner@mcrel.org. 


25 Please note that Institutional Review Boards (IRBs) do have the authority to inspect consent records and data files only to 
assure compliance with approved procedures. If, during the process of collecting data, a threat of violence against an individual 
or entity is uncovered, McREL cannot guarantee anonymity or confidentiality to any party involved. 
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Electronic Consent* 


If you agree to participate in this survey, please click “Yes, I agree” below. By doing so, you are 
indicating that you have read the information on this page, are at least 18 years of age, and that you 
voluntarily agree to participate in the survey. 

If you decline to participate in the survey, click “No, I do not agree” below. 

Yes, I agree to participate in the survey. 

H. No, I do not agree to participate in the survey. 
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General Professional Learning Community (PLC) Information 


1. Please rank the following goals of PLCs in order of importance with 1 being least 
important and 3 being most important* 


Item 

Least important 

1 ! 

2 

Most important 

! 3 

a. 

Improve instmction and assessment practices 

n 

& 

i 

b. 

Diagnose and improve student proficiency 

n 

& 

◄ 

c. 

Develop effective classroom environments to promote 
and support learning 

n 

& 

◄ 


2. Please rank the following PLC practices in order of importance with 1 being least 
important and 3 being most important* 


Item 

Least important 

1 j 

2 

Most important 

1 3 

a. Planning for formative assessment 

n 

& 

◄ 

b. Examining formative work 

□ 

& 

i 

c. Adjusting practice through data 

n 

& 

◄ 


3. According to the PLC Quick Start Guide, there are four types of alignment upon which 
PLCs may focus. Please indicate the level of importance placed upon each type of 
alignment across the PLCs in your school with 1 being least important and 4 being most 
important* 


Item 

Least important 
1 | 2 

Most important 

3 1 4 

a. Aligned learning environment goals 

□ 

& 

i ► 

b. Aligned assessment goals 

n 

& 

i ► 

c. Aligned curriculum goals 

n 

& 

i ► 

d. Aligned instmction goals 

n 

& 

i ► 


4. Are you familiar with the PLC Scoring Guide?* 

li. Yes 

No 
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5. Who uses the PLC Scoring Guide at your school? 

[ This question will onlj appear if ‘Yes” is selected on Q4] 

kk Principal 
kk Assistant principal 
kk Department chairs 
kk Teacher leaders 
kk Teachers 

kk Other (please specify: ) 

6. How frequently is the PLC Scoring Guide used? 

[ This question will only appear if ‘Yes” is selected on Q4J 

kk Daily 
XL Weekly 

kk Two or three times a month 
kk Monthly 
kk Once a trimester 
XL Never 

kk Other (please specify: ) 

7. Do PLC teams observe other PLC teams in the school?* 

Xl Yes 

kk No 

8. Do PLC teams observe PLC teams at other schools?* 
kk Yes 

kk No 

9. There are three “Big Ideas” with corresponding indicators associated with the PLCs. 
Thinking about the PLCs across your school, please rate your level of agreement with 
the indicators for Big Idea HI: Building a Collaborative Culture* 


Indicators 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. 

Collaboration focuses on the group 
learning together 

kk 

XL 

kk 

XL 

b. 

The team is focused on critical questions 

XL 

XL 

kk 

kk 

c. 

Products of collaboration efforts (e.g., 
norms, goals, outcomes, assessments, and 
lesson plans) are explicit 

XL 

XL 

kk 

XL 
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10. Thinking about the PLCs across your school, please rate your level of agreement with 
the indicators for Big Idea #. '2. Ensuring that All Students Learn * 



Indicators 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. 

Identifies/discusses essential common 

.a. 

XX 

XX 

XX 


outcomes 

b. 

Outcomes are aligned with content 
standards/pacing guides 

XX 

XX 

XX 

XX 

c. 

Instructional plans include specific learning 
targets 

XX 

XX 

XX 

XX 

d. 

Teams develop multiple common 
formative assessments/tasks 

XX 

XX 

XX 

XX 

e. 

Instructional plans and activities include 
strategies to increase the use of higher 
order thinking skills 

XX 

XX 

XX 

XX 

f. 

Instructional plans provide support for 
students at different learning levels 

XX 

XX 

XX 

XX 


11 . 


12 . 


Thinking about the PLCs across your school, please rate your level of agreement with 
the indicators for Big Idea #, 3 : Establishing a Focus on Results.* 


Indicators 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Pursues specific and measurable team 
performance goals (SMART goals) 

.a. 

XX 

XX 

XX 

Accesses relevant student information to 

monitor progress toward goals 

XX 

XX 

XX 

XX 

Utilizes common assessments and 

formative tasks to monitor student 
progress 

XX 

XX 

XX 

XX 


d. Reflects on data to identify students in 

need of intervention ('enrichment) and 

V 7 ^ ^ ^ 

plans for strategies, activities, and 

interventions 

e. Utilizes an increasingly directive, timely, 
and systematic response to students not 

: . r r\ r\ r\ 

meeting established outcomes and learning 
targets 

Do staff members in your school use Learning Walks to observe classrooms?* 
XX Yes 

XX No 
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13. Who uses Learning Walks at your school? 

[ This question will onlj appear if ‘Yes” is selected on Q1 2] 

kk Principal 
kk Assistant principal 
kk Department chairs 
kk Teacher leaders 
kk Teachers 

kk Other (please specify: ) 


Content-Based PLCs 


14. How often do content-based PLCs meet at your school?* 

XL Daily 
XL Weekly 

kk Two or three times a month 
kk Monthly 
kk Once a trimester 
XL Never 

kk Other (please specify: ) 

15. For how long do the content-based PLCs meet (i.e., minutes per session)?* 

kk 30 minutes 
Ik 45 minutes 
Ik 60 minutes 
kk 75 minutes 
Ik 90 minutes 
kk 120 minutes 

kk Other (please specify: ) 

16. How many content-based PLCs meet at your school during the 
2014-2015 school year?* 

17. Who leads or facilitates the content-based PLCs (i.e., what is that person’s title)? 



18. In two or three sentences, describe how content-based PLCs are monitored at your 
school.* 
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19. What are two or three benefits you have observed from the content-based PLCs? 

(If you only have two responses, mark “NA ’for the third benefit. )* 


Benefit 1: 
Benefit 2: 
Benefit 3: 




21. What other support or resources do you need to continue to build and improve content- 
based PLCs in your school? 


Cross-Content PLCs 


22. Which type(s) of cross-content PLCs do you have at your school?* 

XX School of Study (SOS) 

XX Math/English/ Social Studies/Science (MESS) 

XX Q4R2 
XX None 

XX Other (please specify: ) 

23. What teachers (i.e., subject areas) are paired? 


24. How often do cross-content PLCs meet at your school? 

XX Daily 
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JX Weekly 

Xi. Two or three times a month 
H. Monthly 
H. Once a trimester 
H. Never 

Xi. Other (please specify: ) 

25. For how long do the cross-content PLCs meet at your school (i.e., minutes per session)? 

H. 30 minutes 
H. 45 minutes 
H. 60 minutes 
75 minutes 
H. 90 minutes 
H. 120 minutes 

a Other (please specify: ) 

26. How many cross-content PLCs meet at your school during the 
2014-2015 school year? 

27. Who leads or facilitates the cross-content PLCs (i.e., what is that person’s title)? 



28. In two or three sentences, describe how cross-content PLCs are monitored at your 
school. 


29. What are two or three benefits you have observed from the cross-content PLCs? 

(if you only have two responses, mark ‘NA ’for the third benefit. ) 


Benefit 1: 


Benefit 2: 


Benefit 3: 


30. What are two or three challenges your school has faced in implementing cross-content 
PLCs? (If you only have hvo responses, mark “NA ’for the third benefit. ) 
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31. What other support or resources do you need to continue to build and improve cross- 
content PLCs in your school? 



32. Does your school have a written protocol in place for determining how freshmen should 
be assigned to courses?* 

Ik Yes 

kk No 

33. Does your school have a written protocol in place for determining how upperclassmen 
should be assigned to courses?* 

kk Yes 

kk No 

34. Does your school have a written protocol or guidelines in place for monitoring student 
progress throughout the year?* 

kk Yes 

kk No 

35. Does your school have a written protocol or guidelines in place for making adjustments 
to student course placement based on student pass/fail rates in courses?* 

kk Yes 

kk No 

36. Has the trimester addressed the needs of struggling students in your school?* 
kk Yes 

kk No 


37. Please describe how the trimester has addressed the needs of struggling students in your 
school. [ This question will only appear if ‘Yes” is selected on Q36] 
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38. Please describe how the trimester has not addressed the needs of struggling students in 
your school. [ This question will only appear if “No” is selected on Q3 6 ] 


39. Has the trimester schedule addressed the needs of advanced or accelerated students?* 

Yes 

No 

40. Please describe how the trimester schedule has addressed the needs of accelerated or 
advanced students, f This question will only appear if ‘Yes” is selected on 039] 


41. Please describe how the trimester schedule has not addressed the needs of accelerated or 
advanced students, f This question will only appear if ‘No” is selected on Q39] 


42. What are two or three factors that have helped address the needs of all students at your 
school? (If you only have two responses, mark ‘NA ’for the third factor. )* 


Factor 1: 
Factor 2: 
Factor 3: 


43. What have been two or three challenges associated with addressing the needs of all 
students at your school? (If you only have two responses, mark ‘NA ’for the third challenge .)* 

Challenge 1: 


Appendix B- 1 0 


Challenge 2: 
Challenge 3: 


44. What additional support or resources do you need to effectively schedule and monitor 
student progress? 



45. In two to three sentences, please describe what you hope the CATs achieve by the end of 
the year.* 


46. Rank the following components of CAT from most important to least important with 
1 being least important and 4 being most important* 


Item 

Least important 

1 | 2 j 

Most important 

3 | 4 

a. Advance college -ready skills, monitor student 





progress, and increase students’ sense of affiliation 
with adults and peers 

n 

& 

◄ 

► 

b. Build knowledge and use of college -ready skills and 
habits 

n 

& 

◄ 

► 

c. Improve students’ sense of affiliation with adults 
and peers 

n 

& 

i 

► 

d. Strengthen peer-to-peer relations and perceptions 
of peer support, particularly within career themes 

n 

& 

◄ 

► 


47. Rank the following based on importance in CAT with 1 being least important and 
4 being most important* 


Item 

Least important 

Most important 


1 | 2 

3 | 4 
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a. Monitor students’ academic progress and intervene ^ 
with subject-specific remediation 

b. Develop career interests and motivation, and 

ensure appropriate, related college-bound course □ 

taking and ACT preparation 

c. Develop 21st century skills including study skills, 
persistence, independence, adaptation to change, 

digital literacy, effective communication, inventive □ 

thinking, and fostering motivation to high 

achievement 

d. Develop knowledge and support for college 
application and financial assistance planning 


& 

& 


& 


& 


i 


i 


i 


i 


► 


► 


► 


► 


48. What are two or three factors that facilitate successful CAT implementation? 

(If you only have hvo responses, mark ‘NA ’for the third factor. )* 


Factor 1: 
Factor 2: 
Factor 3: 


49. What are two or three challenges you have faced in implementing CAT at your school? 

(If yon only have hvo responses, mark “NA ’for the third challenges .)* 


Challenge 

Challenge 

Challenge 



Thank you for completing this survey. We appreciate your time and effort. 
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Spring 20 1 5 College Access Resource Teacher (CART) 
Implementation Survey 


Jefferson County Public Schools (JCPS) 

Investing in Innovation (i3) 

College Access Resource Teacher (CART) Survey 

Spring 20 1 5 

McREL International (formerly the Mid-Continent Regional Education Laboratory and Edvantia, 
Inc.), an education research company, is working with the Jefferson County Public Schools (JCPS) 
district to evaluate the Investing in Innovation (i3) project. This survey is being conducted as part 
of the i3 grant evaluation with a specific focus on how College Advisory Time (CAT) is being 
implemented at your school. Project staff want to know about your thoughts, experiences, and so 
on related to your involvement in the implementation of CAT so they can use that information to 
make adjustments and improvements. Data from this survey will be summarized in a report 
provided to JCPS project staff. The survey should take approximately 25 to 30 minutes to complete. 
Please complete by Friday, May 1, 2015. 

Protecting Your Rights: Participation in this survey is voluntary and should not involve any 
known risks above those normally encountered in daily life. There will be no retribution of any sort 
for the responses you provide. If you feel uncomfortable about responding to some questions, 
please feel free to skip those questions. You may also choose to stop participating in the survey 
altogether without penalty or reprisal. 

About Your Confidentiality: Survey responses will be collected via Qual tries utilizing transport 
layer security (TLS) encryption. TLS encryption is a security feature that encrypts the survey link 
and data as it moves across the Internet. McREL evaluators will handle the data and do everything 
they can to ensure its security 26 . Although most responses will be aggregated across schools, please 
note that your responses may be identified by school. This will help us to identify practices being 
implemented at schools that may warrant further study. 

Benefits: Although there are no direct personal benefits for you and you will not receive any 
compensation for participating in the survey, the information you provide will give project staff 
better information about how to improve the project. These kinds of improvements may result in 
indirect benefits to you as a participant in the project. 

Questions: If you have any questions about this survey, please contact Dr. Tara Donahue, 
Managing Evaluator (800.624.9120 ext. 5419; tdonahue@mcrel.org). For information on protection 
of your rights as a participant, contact Karen Bumgardner, a member of McREL’ s Institutional 
Review Board, at 800.624.9120, ext. 5841, or kbumgardner@mcrel.org. 


26 Please note that Institutional Review Boards (IRBs) do have the authority to inspect consent records and data files only to 
assure compliance with approved procedures. If, during the process of collecting data, a threat of violence against an individual 
or entity is uncovered, McREL cannot guarantee anonymity or confidentiality to any party involved. 
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Electronic Consent* 


If you agree to participate in this survey, please click “Yes, I agree” below. By doing so, you are 
indicating that you have read the information on this page, are at least 18 years of age, and that you 
voluntarily agree to participate in the survey. 

If you decline to participate in the survey, click “No, I do not agree” below. 

Yes, I agree to participate in the survey. 

H. No, I do not agree to participate in the survey. 
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General Questions about CAT 


1. In two to three sentences, please describe a typical College Access Time (CAT) or 
advisory period in your school for each grade level.* 



2. In two to three sentences, please describe what you hope to achieve during CAT by 
the end of the year.* 


3. Rank the following components of CAT from most important to least important with 
1 being least important and 4 being most important* 


Item 

Least important 
1 | 2 ; 

Most important 

3 | 4 

a. Advance college -ready skills, monitor student 





progress, and increase students’ sense of affiliation 
with adults and peers 

n 

& 

i 

► 

b. Build knowledge and use of college -ready skills 
and habits 

□ 

& 

◄ 

► 

c. Improve students’ sense of affiliation with adults 
and peers 

n 

& 

◄ 

► 

d. Strengthen peer-to-peer relations and perceptions 
of peer support, particularly within career themes 

n 

& 

◄ 

► 
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Design Team 


4. Do you have a CAT design team?* 

LL Yes 

XL No l Qualtrics will skip to Q? since the following questions are not applicable.] 

5. How many members are on the design 
team? 

6. How often does the design team meet? 

XL Daily 
-CL Weekly 

XL Two or three times a month 
XL Monthly 
XL Once a trimester 
XL Once a year 
XL Never 

XL Other (please specify: ) 


7. In two to three sentences, describe the purpose of the design team. 


8. What topics are discussed in the design team? 


Topic 

Topic 

Topic 

Topic 

Topic 
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CAT Content 


9. Approximately what percentage of CAT is devoted to the following? (Total must sum 
to 100%.)* 


Items 

a. Monitor students’ academic progress and intervene with subject-specific 
remediation 


% 


b. Develop career interests and motivation, and ensure appropriate, related college- 
bound course taking and ACT preparation 

c. Develop 21st century skills including study skills, persistence, independence, 
adaptation to change, digital literacy, effective communication, inventive thinking, 
and fostering motivation to high achievement 

d. Develop knowledge and support for college application and financial assistance 
planning 

Total 


10. Do you use the walkthrough instrument to monitor CAT implementation?* 
XL Yes 

XL No 


11. How often do you use it? [ This question will only show if ‘Yes” is selected forQIO] 

XL Weekly 

XL Two or three times a month 
XL Monthly 
XL Once a trimester 
XL Twice a year 
LL Once a year 

12. Do you use any other tool to monitor CAT implementation? Please describe. 

[ This question will only show if “No” is selected forQIO] 


13. How often do CAT lessons follow the elements of the Classroom Instructional 
Framework? 


Items 

Seldom 

Sometimes 

Almost 

always 

a. Establishing engagement 

XL 

XL 

XL 

b. Fostering connections 

XL 

XL 

XL 

c. Deepening understanding 

XL 

XL 

XL 

d. Making meaning 

XL 

XL 

XL 
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14. What have been two or three factors that have made CAT implementation successful 
in your school during the 2014-2015 school year? (If you only have two responses, mark ‘NA” for 
the third factor. )* 


Factor 1: 
Factor 2: 
Factor 3: 


15. What have been your two or three biggest challenges? ( If you only have two responses, mark 
‘NA ’’for the third challenge .)* 



Advisor Training 


16. Did you train the advisors in your school for the 2014-2015 school year?* 
Yes 

.a. No 


17. How many hours of training did you provide? 

[ This question will only show if ‘Yes” is selected forQ16] 

18. Why not? (This question will only show if “No ” is selected for Q16( 


19. What are two or three factors that have facilitated advisor effectiveness? 


Factor 

Factor 

Factor 
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20. What are two or three challenges advisors face? 



CAT Logistics 


21. How often does your CAT period meet?* 

li. Daily 

Weekly 

Two or three times a month 
Monthly 

a Other (please specify: ) 

22. How many weeks does your CAT period meet during the school year?* 

Xi. Fewer than 15 weeks 
1 5 to 1 9 weeks 
H. 20 to 24 weeks 
n. 25 to 29 weeks 
n. 30 to 34 weeks 
More than 34 weeks 

a Other (please specify: ) 

23. How many minutes per week does your CAT period meet?* 

H. 30 minutes 
H. 45 minutes 
55 minutes 
H. 60 minutes 
75 minutes 
H. 90 minutes 

a Other (please specify: ) 


24. What is your ratio of students to teachers during CAT time?* 
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-CL 1:10 
-£1 1:15 
-CL 1:18 

-CL 1:20 

-CL 1:25 

XX Other (please specify: ) 

25. How are students assigned to CATs?* 

ix Grade level 
XX School of Study 
XX Student interest 
XX Gender 

XX Other (please specify: ) 

26. How many school days prior to CAT do advisors receive their lessons?* 

XX 1 day 

XX 2 days 
XX 3 days 
XX 4 days 
XX 5 days 

XX Other (please specify: ) 

27. How often do students provide formal feedback about CAT?* 

XX Daily 
XX Weekly 

XX Two or three times a month 
XX Monthly 
XX Never 

XX Other (please specify: ) 

28. Describe how you gather feedback from students. 
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CART Training 


29. How many CAT training sessions provided by JCPS did you attend during the 2014-2015 
school year?* 

XL 1 
XL 2 
xl 3 
XL 4 
Xl 5 

Xl Other (please specify: ) 

30. What topics were covered during your training sessions? 


Topic 1: 


Topic 2: 


Topic 3: 


Topic 4: 


Topic 5: 


Topic 6: 


Topic 7: 


Topic 8: 


Topic 9: 


Topic 10: 


31. Did you find the topics covered during the training helpful in implementing an advisory 
period in your school? 

H. Yes 

XL No 


32. What topics would be useful for you? [ This question will only show if “No” is selected for Q30 ] 
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Topic 1: 


Topic 2: 

Topic 3: 

Topic 4: 

Topic 5: 

33. Please indicate your level of agreement with the following statements:* 


Items 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. 

The training sessions provided by JCPS are 
valuable. 

XX 

XX 

XX 

XX 

b. 

The SharePoint resources are valuable. 

XX 

XX 

XX 

XX 

c. 

The coaching sessions are valuable. 

XX 

XX 

XX 

XX 


34. What other training, support, or resources do you need for successful advisory 
implementation in your school? 


Final Comments 


35. Please include any other comments or observations you have about CAT 
implementation at your school. 


Thank you for completing this survey. We appreciate your time and effort. 
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20 1 5 Student Survey 


Appendix C 


20 1 5 Student Survey 


(^McREL 

yC")/ INTERNATIONAL 


2015 JCPS i3 Student Survey 


We would like to learn how you feel about your experiences in the classroom and your school in general. We are going to ask 
your thoughts about the difficulty of your courses, how you see yourself as a student, your College Advisory/ Access Time (CAT), 
and how your courses meet your needs as a student Your responses are anonymous; this means that your name will not be 
linked to your survey. You will not be identified or named in any reports. Your answers to this survey will never be given to 
your parents (or guardians), your teachers, or anyone other than the researchers who will wrice the report about the project 


There are no right or wrong answers — choose the answer that is right for you. It is important that you answer the questions 
based on what you really do or know. If you do not want to answer a question, just leave it blank. Your participation is 
voluntary; you decide whether or not to take the survey. If you do not wish to participate in the survey, curn in your blank 
survey without penalty. Your completed (or blank) survey will be sealed in an envelope and sent to the researchers. Thank you 
for your help! 


With PENCIL or BLUE or BLACK INK, please fill in the circle that most closely corresponds to your response for each of the 
following questions/statements. 

Please mark responses like this: ® Not like this: 2) or 0 







Based on your experiences during the CURRENT trimester, 
please tell us how much you agree with the following sentences. 

Not 
Taking 
This Class 

Disagree 
A Lot 

Disagree 
A Little 

Agree 
A Little 

Agree 
A Lot 

1 . My English class motivates me to learn more about English. 

O 

0 

© 

© 

© 

2. 1 feel like my English class is too easy for me. 

O 

O 

© 

© 

© 

3. 1 feel like my English class is too hard for me. 

o 

© 

© 

© 

© 

A. My math class motivates me to learn more about math. 

o 

© 

© 

© 

© 

5. 1 feel like my math class is too easy for me. 

o 

© 

© 

© 

© 

6. 1 feel like my math class is too hard for me. 

o 

© 

© 

© 

© 

7. My science class motivates me to learn more about science. 

o 

o 

© 

© 

© 

8. 1 feel like my science class is coo easy for me. 

o 

o 

© 

© 

© 

9. 1 feel like my science class is too hard for me. 

o 

© 

© 

© 

© 

IQ My social studies class motivates me to learn more about 
social studies. 

o 

© 

© 

© 

© 

II. 1 feel like my social studies class is too hard for me. 

o 

© 

© 

© 

© 

12. 1 feel like my social studies class is too hard for me. 

o 

© 

© 

© 

© 

Please tell us how much you agree with the following sentences. 


Disagree 
A Lot 

Disagree 
A Little 

Agree 
A Little 

Agree 
A Lot 


13. 1 am not doing well in school. 


© 

© 

© 

© 

1 ^ My school emphasizes spending a significant amount of time on 
school work and studying. 


o 

© 

© 

© 

1 S. 1 think I'm a good student 


© 

© 

© 

© 

How sure are you that things will work out well 

Not Sure 
At All 

Mostly 
Not Sure 

Mostly 

Sure 

Very Sure 

1 6. when you have co learn something new at school? 


© 

© 

© 

© 

1 7. when you have co give a talk in class? 


© 

© 

© 

© 

1 8. when you have co do an accivicy for che first time? 


O 

© 

© 

© 


© 2015 McREL International 


PLEASE CONTINUE 
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How sure are you that things will work out well . . . 


Not Sure 
At All 

Mostly 
Not Sure 

Mostly 

Sure 

Very Sure 

1 9. when you are having trouble with your schoolwork? 


O 

© 

© 

© 

20. when you have to figure out something by yourself? 


© 

© 

© 

© 

CAT has helped me . . . 

Have Not 
Covered 
Topic 

Disagree 
A Lot 

Disagree 
A Little 

Agree 
A Little 

Agree 
A Lot 

21. understand the college admission process. 

O 

© 

© 

© 

© 

22. prepare for the PLAN® and/or ACT® test 

O 

© 

© 

© 

© 

22 develop life skills (e.g., time management communication, conflict 
resolution, and financial literacy). 

o 

© 

© 

© 

© 

During my past and current experiences in CAT . . . 

Have Not 
Covered 
Topic 

Disagree 
A Lot 

Disagree 
A Little 

Agree 
A Little 

Agree 
A Lot 

24. 1 have had opportunities to research and develop career interests. 

o 

© 

© 

© 

© 

25. 1 have learned how to complete a college application. 

o 

© 

© 

© 

© 

2 £ 1 have learned about financial literacy, financial aid, and paying for 
college. 

o 

© 

© 

© 

© 

27. 1 have created a resume. 

o 

© 

© 

© 

© 

28. 1 have learned how to apply for a job. 

o 

© 

© 

© 

© 

1 feel 1 can go to . . . 


Disagree 
A Lot 

Disagree 
A Little 

Agree 
A Little 

Agree 
A Lot 

29. my CAT advisor when 1 need help with my classes. 


© 

© 

© 

© 

30. my CAT advisor when 1 have a personal problem. 


0 

© 

© 

© 

3 1. students in my CAT when 1 have an academic problem. 


© 

© 

© 

© 

32. students in my CAT when 1 have a personal problem. 


© 

© 

© 

© 

In general, . . . 


Disagree 
A Lot 

Disagree 
A Little 

Agree 
A Little 

Agree 
A Lot 

3 j if 1 am struggling in a class, 1 have the opportunity to change to a 
different course. 


© 

© 

© 

© 

24 if 1 do not pass a class, there are opportunities for me to make up the 
credit for that class. 


© 

© 

© 

© 

32 1 have opportunities to take classes outside my core classes 
(e.g., foreign language, music, art and theater). 


© 

© 

© 

© 

36. 1 have opportunities to take honors or advanced classes. 


© 

© 

© 

© 


37. In what grade are you? 39. Which race/ethnicity best describes you? 40. What is your G PA? 


o 

Freshman (9th) 

o 

Asian 

o 

4.01 or above 

o 

Sophomore (10th) 

o 

Black/African American 

o 

3.51 -4.00 

o 

Junior (1 Ith) 

o 

Hispanic/Latino 

o 

3.01 -3.50 

o 

Senior (12th) 

o 

White/Caucasian 

o 

2.51 -3.00 



o 

Multiple ethnicity/Other 

o 

2.01 -2.50 

What 

is your sex? 


(please specify): 

o 

1.51 -2.00 

o 

Male 



o 

1.01 - 1.50 

o 

Female 



o 

1.00 or below 





o 

I'm not sure 
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Appendix D 


20 1 5 Teacher Survey 


Investing in Innovation (i3) Grant Implementation Evaluation: 

Making Time for What Matters Most 

Spring 20 1 5 Teacher Survey - Consent Form 

In October 20 1 0, Jefferson County Public Schools (JCPS) was awarded a four-year development grant by 
the U.S. Department of Education through the inaugural Investing in Innovation (i3) program 
competition. JCPS’ project, Making Time for What Matters Most, aims to improve student achievement, 
narrow achievement gaps, strengthen students’ college readiness skills, and increase the percentage of 
students who graduate and go on to college. Interventions include trimester scheduling, college access 
time/advisory, and teacher professional learning communities. 

Purpose of the Survey: JCPS’ i3 grant program is being evaluated - meaning that it is being looked at 
and studied to see if it is doing what it is supposed to do. McREL International, an independent research 
firm, is working with JCPS to figure out whether the i3 grant is achieving its goals. To help with this 
evaluation, we are surveying all core content teachers at the participating schools to ask about your 
thoughts and experiences related to your involvement in the i3 project (e.g., trimester scheduling, 
college access time/advisory, and professional learning communities). McREL is not evaluating the 
effectiveness of your school or individual teachers. Rather, the data obtained from the survey will be 
reported in an aggregated form to assist JCPS in making project-wide adjustments and improvements. 

Protecting Your Rights: Participation in this survey is voluntary and should not involve any known 
risks above those normally encountered in daily life. There will be no retribution of any sort for the 
responses you provide. If you feel uncomfortable about responding to some questions, please feel free 
to skip those questions. You may also choose to stop participating in the survey altogether without 
penalty or reprisal. 

About Your Confidentiality: Survey responses will be collected via Qualtrics utilizing transport layer 
security (TLS) encryption. TLS encryption is a security feature that encrypts the survey link and data as 
it moves across the Internet. McREL evaluators will handle the data and do everything they can to 
ensure its security 27 . Although most responses will be aggregated across schools, please note that your 
responses may be identified by school. This will help us to identify practices being implemented at 
schools that may warrant further study. 

Benefits: While there are no direct benefits to individuals for participating, the survey is one of several 
data sources that we will use to evaluate JCPS’ i3 grant program. Our evaluation report will provide 
crucial information to JCPS to help them understand the impact of the i3 grant on various stakeholders 
and improve specific reform initiatives. In appreciation of your time and contribution, you will receive a 
professional development stipend through JCPS. 

Questions: If you have any questions about this survey, please contact Dr. Tara Donahue, Managing 
Evaluator (800.624.9120 ext. 5419; tdonahue@mcrel.org). For information on protection of your rights 
as a participant, contact Karen Bumgardner, a member of McREL’s Institutional Review Board, at 
800.624.9120, ext. 5841, or kbumgardner@mcrel.org. 


27 Please note that Institutional Review Boards (IRBs) do have the authority to inspect consent records and data files only to 
assure compliance with approved procedures. If, during the process of collecting data, a threat of violence against an individual 
or entity is uncovered, McREL cannot guarantee anonymity or confidentiality to any party involved. 
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Electronic Consent* 


If you agree to participate in this survey, please click “Yes, I agree” below. By doing so, you are 
indicating that you have read the information on this page, are at least 18 years of age, and that you 
voluntarily agree to participate in the survey. 

If you decline to participate in the survey, click “No, I do not agree” below. 
xx Yes, I agree to participate in the survey. 
xx No, I do not agree to participate in the survey. 


Appendix D-2 



Investing in Innovation (i3) Grant Implementation Evaluation 

Teacher Survey - Spring 2015 


Instructions: Please answer the following questions based on your experiences during the current 
school year (2014-2015). 


I. At which i3 school do you teach? 
xx Academy @ Shawnee 
xx Fern Creek Traditional High School 
xx Moore Traditional High School 
xx Valley Traditional High School 
xx Waggener High School 
xx Western High School 

Section I: Teacher Collaboration and Support 

Please tell us about your experiences with teacher collaboration and support at your school. 


2. Rate the extent to which you agree or disagree with each of the following statements. 


Statements 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. Teachers have sufficient time to collaborate with colleagues to plan and 
improve instruction. 

n 

& 

i 

► 

b. Collaboration is seen as an important part of this school's culture. 

n 

& 

i 

► 

c. Teachers have sufficient resources to provide quality instruction. 

n 

& 

◄ 

► 

d. Teachers are generally willing to try new ideas. 

n 

& 

i 

► 

e. Teachers in this school trust each other. 

n 

& 

◄ 

► 

f. It is okay in this school to discuss with other teachers positive and/or 
negative feelings, concerns, and frustrations related to instruction and 
practice. 

n 

& 

◄ 

► 

g. Teachers discuss difficulties and work together to develop solutions. 

n 

& 

◄ 

► 

h. Teachers support each other's efforts to make improvements. 

n 

& 

◄ 

► 

i. Access to expertise and resources is available and provided in a timely 
manner. 

n 

& 

◄ 

► 


Section II: Professional Learning Communities (PLCs) 

Please tell us about your experiences with PLCs at your school. 


3. In general, who leads or facilitates PLCs at your school? Select all that apply. 

Principal ' Teachers 

Assistant Principal ' Educational Recovery Specialist or Interventionist 

Department Chair ' I don't know 

Teacher Leaders ' Other (please specify): 
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4. In thinking about the PLCs at your school, please rank the following PLC goals in order from most to least 
important with I being least important and 3 being most important. 


PLC Goals 

I st 

2nd 

3rd 

a. Improve instruction and assessment practices 

□ 

0 

4 

b. Diagnose and improve student proficiency 

□ 

0 

4 

c. Develop effective classroom environments to promote and support 
learning 

□ 

0 

4 

In thinking about the PLCs at your school, please rank the following PLC practices in 
important with 1 being least important and 3 being most important. 

order from most to least 

PLC Practices 

I st 

2nd 

3rd 

a. Planning for formative assessment 

□ 

0 

4 

b. Examining formative work 

□ 

0 

4 

c. Adjusting practice through data 

□ 

0 

4 


6. In thinking about the PLCs at your school, please rank the following PLC topics covered in order from most to 
least important with I being least important and 4 being most important. 


PLC Topics Covered 

I st 

2nd 

3rd 

4th 

a. Assessment 

□ 

0 

◄ 

► 

b. Curriculum 

□ 

0 

4 

► 

c. Instruction 

□ 

0 

4 

► 

d. Learning environment 

□ 

0 

4 

► 

Rate the extent to which you agree or disagree with each of the following statements. 

Statements 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. 1 feel that my contribution to the PLC is valued. 

□ 

0 

4 

► 

b. Successful results with students are a regular part of discussions during 
PLCs. 

□ 

0 

4 

► 

c. Teacher collaboration related to instruction has increased as a result of 
my school’s PLCs. 

□ 

0 

4 

► 

d. The PLC facilitator is adequately prepared for all PLC meetings. 

□ 

0 

4 

► 

e. School administrators (e.g., principal and assistant principal) attend the 
PLC meetings. 

□ 

0 

4 

► 

f. 1 feel that my instructional practice has improved as a result of 
participating in PLCs. 

□ 

0 

4 

► 

g. Time for collaborative planning is uninterrupted. 

□ 

0 

4 

► 

h. New ideas are welcomed and supported. 

□ 

0 

4 

► 

i. The PLC facilitator encourages all voices to be heard. 

□ 

0 

4 

► 
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PLC Participation: Course-alike / Content-based 


8. Please indicate the course-alike / content-based PLC of which you are a member. Select all that apply. 


English Dept. 
English I 
English 2 
English 3 
English 4 


Math Dept. 
Algebra I 
Algebra 2 
Applied Math 
Geometry 


Science Dept. 

Biology 

Integrated Science I A 
Integrated Science I B 
Other (please specify): 


Social Studies Dept. 
Humanities and Related Arts 
U.S. History 
World Civics 


Not applicable - I do not participate in a course-alike/content-based PLC (skip to question #11) 


9. In general, how often do course-alike / content-based PLCs meet at your school? 

Daily ' Once a trimester (3 times per year) 

Weekly ' Once or twice per year 

Two to three times per month ' Never 

Monthly ' I don't know 

Other (please specify): 


10. On average, how long does a course-alike / content-based PLC meeting last? 
30 minutes ' 70 minutes 

45 minutes ' 90 minutes 

60 minutes ' I don’t know 

Other (please specify): 


PLC Participation: Cross-content / School of Study (SOS) 

I I . Please indicate the cross-content / SOS PLC of which you are a member. Select all that apply. 

Analytical and Applied Sciences (SOS) ' Freshman Academy - Grade Level 

Environmental Sciences (SOS) ' Sophomore - Grade Level 

Health Sciences (SOS) ' Junior - Grade Level 

Medical Sciences (SOS) ' Senior - Grade level 

Communication and Media (SOS) ' Math/English/Science/Social Studies (MESS) 

Leadership and Social Sciences (SOS) ' Q4R2 (Quadrant 4, Higher Rigor and Relevance Challenge Level) 

Visual and Performing Arts (SOS) ' Other (please specify): 

Not applicable - I do not participate in a cross-content or SOS PLC (skip to question #14) 
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12. In general, how often do cross-content / SOS PLCs meet at your school? 

Daily ' Once a trimester (3 times per year) 

Weekly ' Once or twice per year 

Two to three times per month ' Never 

Monthly ' I don't know 

Other (please specify): 


On average, how long does a cross-content / SOS PLC meeting last? 

30 minutes ' 70 minutes 

45 minutes ' 90 minutes 

60 minutes ' I don’t know 

Other (please specify): 


PLC Big Ideas 

There are three "Big Ideas" with corresponding indicators associated with PLCs. Each indicator is listed below. 

14. Rate the extent to which you agree or disagree with the indicators for Big Idea #/: Building a Collaborative Culture. 


Indicators 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. Collaboration focuses on the group learning together. 

□ 

0 

◄ 

► 

b. The PLC team is focused on critical questions. 

□ 

0 

4 

► 

c. Products of collaboration efforts (e.g., norms, goals, outcomes, 
assessments, and lesson plans) are explicit. 

□ 

0 

4 

► 

Rate the extent to which you agree or disagree with the indicators for Big Idea #2: Ensuring that All Students Learn. 

Indicators 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. The PLC identifies/discusses essential common outcomes. 

□ 

0 

4 

► 

b. Outcomes align with content standards/pacing guides. 

□ 

0 

4 

► 

c. Instructional plans include specific learning targets. 

□ 

0 

4 

► 

d. PLC teams develop multiple common formative assessments/tasks. 

□ 

0 

4 

► 

e. Instructional plans and activities include strategies to increase the use 
of higher order thinking skills. 

□ 

0 

4 

► 

f. Instructional plans provide support for students at different learning 
levels. 

□ 

0 

4 

► 
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1 6. Rate the extent to which you agree or disagree with the indicators for Big Idea #3: Establishing a Focus on Results. 


Indicators 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. PLCs pursue specific and measurable team performance goals 
(SMART goals). 

□ 

0 

4 

► 

b. PLCs access relevant student information to monitor progress 
toward goals. 

□ 

0 

4 

► 

c. PLCs utilize common assessments and formative tasks to monitor 
student progress. 

□ 

0 

4 

► 

d. PLCs reflect on data to identify students in need of intervention 
(enrichment) and plans for strategies, activities, and interventions. 

□ 

0 

4 

► 

e. PLCs utilize an increasingly directive, timely, and systematic response 
to students not meeting established outcomes and learning targets. 

□ 

0 

4 

► 


Section III: Course Assignment Process and Trimester Schedule 

Thinking about your experiences with the course assignment process and trimester scheduling, please share those 
experiences with us by answering the questions below. 

17. Indicate the extent to which you agree or disagree with the following statements about your school’s 
course assignment process. 


My school . . . 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

has a course assignment process that results in the best placement for 
students. 

n 

0 

4 

► 

has a clear process for identifying students early when they are struggling. 

n 

0 

◄ 

► 

has effective practices in place to successfully intervene with students 
who are struggling in a course. 

has effective practices in place to move students up to a higher level 
course when they have been successful. 

□ □ 

0 

0 

◄ 

◄ 

► 

► 


1 8. Has the trimester schedule addressed the needs of struggling students in your school? 

V Yes <$> No 

1 9. Has the trimester schedule addressed the needs of advanced or accelerated students in your school? 

V Yes <a> No 
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Section IV: College Access Time (CAT) 

Please tell us about your school’s CAT or advisory, a designated time for student college preparation and advisory. 


20. Rank the following components of CAT in order of importance with I being least important and 4 being most 
important. 


CAT Components 

I st 

^nd 

3 rd 

4 th 

a. Advance college-ready skills, monitor student progress, and increase 
students' sense of affiliation with adults and peers. 

n 

& 

◄ 

► 

b. Build knowledge and use of college-ready skills and habits. 

n 

& 

i 

► 

c. Improve students' sense of affiliation with adults and peers. 

n 

& 

i 

► 

d. Strengthen peer-to-peer relations and perceptions of peer support, 
particularly within career themes. 

n 

& 

◄ 

► 

Do you lead a CAT advisory period? 

V Yes <®> No (skip to question # 27) 





Approximately how many students are in your CAT advisory period? 
Students (#): 





How many school days prior to your CAT advisory period do you receive your lesson? 

1 day ' 4 days 

2 days ' 5 days 

3 days ' Other (please specify): 




Please rank the following CAT focus areas from most to least important with 
the most important. 

1 being the least important and 4 being 

CAT Focus Areas 

I st 

^nd 

3 rd 

4 th 

a. Monitor students' academic progress and intervene with subject- 
specific remediation. 

n 

& 

i 

► 

b. Develop career interests and motivation, and ensure appropriate, 
related college-bound course taking and ACT preparation. 

n 

& 

◄ 

► 

c. Develop 21st century skills including study skills, persistence, 
independence, adaptation to change, digital literacy, effective 
communication, inventive thinking, and fostering motivation to high 
achievement. 

n 

& 

◄ 

► 

d. Develop college knowledge and support for application and financial 
assistance planning. 

n 

& 

◄ 

► 
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25. How often do CAT lessons follow the elements of the Classroom Instructional Framework? 


Elements 

Never 

Seldom Sometimes 

Almost 

always 

a. Establishing engagement 

n 

& 

◄ 

► 

b. Fostering connections 

n 

& 

◄ 

► 

c. Deepening understanding 

□ 

& 

◄ 

► 

d. Making meaning 

□ 

& 

i 

► 

Indicate the extent to which you agree or disagree with the following statements about your school’s CAT 

or 

advisory time. 





Statements 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. 1 have received sufficient training and coaching to successfully 
implement CAT. 

n 

& 

◄ 

► 

b. The CAT lessons provided to me can be sufficiently covered during 
the allotted time. 

n 

& 

◄ 

► 

c. 1 feel equipped to address social and emotional issues that arise during 
CAT. 

n 

& 

i 

► 

d. 1 believe that the CAT advisory periods are beneficial to students. 

n 

& 

◄ 

► 

e. 1 receive the CAT lessons in time to adequately prepare for the 
lesson. 

□ 

& 

◄ 

► 

f. CAT advisory periods increase students’ college readiness. 

n 

& 

◄ 

► 

g. Overall, CAT advisory periods are making a positive contribution to 
the success of our students. 

n 

& 

◄ 

► 


Section V: Teacher Self-Efficacy 

Thinking about your teaching experiences this year, please share your thoughts on your sense of confidence to deal 
with all the demands and challenges that are implied in teachers’ professional life by answering the questions below. 

27. Indicate how often the following situations have occurred. 


How often . . . 

Never 

Seldom 

Sometimes 

Almost 

always 

do you feel that you are able to work effectively? 

n 

& 

i 

► 

are you satisfied with the quality of your work? 

n 

& 

i 

► 

do you feel that you are being successful in your work? 

n 

& 

◄ 

► 

do you have sufficient self-confidence to defend your own points of 
view? 

n 

& 

i 

► 

do you feel adequately prepared to use formative data (e.g., class 
work, homework, and other instructional activities) to adjust your 
instruction? 

n 

& 

i 

► 

do you feel adequately prepared to use summative data (e.g., end of 
chapter, unit, or course assessments) to adjust your instruction? 

n 

& 

i 

► 

do you feel confident in teaching all required content standards in your 
subject area? 

n 

& 

i 

► 
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How often . . . 

Never 

Seldom Sometimes 

Almost 

always 

do you feel confident in adjusting your instructional practices to meet 
the needs of individual students? 

□ 

& 

i 

► 

do you worry about being criticized if positive results are not readily 
apparent? 

n 

& 

◄ 

► 

does the emphasis on success discourage you from trying new 
approaches? 

n 

& 

i 

► 

action VI: Student Academic Engagement and Challenge 

ease tell us about your teaching experiences this year as it relates to student academic engagement and challenge. 

5. Indicate the extent to which you agree or disagree with the following statements. 




Teachers at my school . . . 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

challenge students academically. 

n 

& 

◄ 

► 

encourage students to go beyond stated expectations. 

n 

& 

◄ 

► 

involve students in setting expectations. 

n 

& 

◄ 

► 

involve students in planning lessons. 

n 

& 

◄ 

► 

involve students in developing criteria for assessing their assignments. 

n 

& 

i 

► 

work with disenfranchised students to help them feel more connected to 
school. 

n 

& 

i 

► 


29. Indicate the extent to which you agree or disagree with the following statements. 




Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

a. 

In general, students seem to enjoy my school. 

n 

& 

◄ 

► 

b. 

High grades at my school are viewed as an indicator of easy course 
work. 

n 

& 

i 

► 

c. 

In general, students seem to be bored at my school. 

n 

& 

i 

► 

d. 

Students at my school perceive challenge as a way to go beyond 
expectations. 

n 

& 

◄ 

► 

e. 

High grades at my school are viewed as an indicator of mastering high 
standards. 

n 

& 

◄ 

► 

f. 

In general, students do well in my school. 

n 

& 

i 

► 


Section VII: Background 

30. What is your highest degree? 3 I . Which grade(s) do you teach? Select all that apply. 


Associate 

Freshman (9th) 

Bachelor 

Sophomore (1 0th) 

Master 

' Junior (Nth) 

Ed.D. 

Senior (12th) 

Ph.D. 



Appendix D- 1 0 






32. How many years have you been teaching? 

This is my first year 
2 to 5 years 
6 to 1 0 years 
I I to 20 years 
2 1 + years 

34. What is your ethnicity/race? Select all that apply. 
American Native/Alaska Native 
Asian 

Black/African American 
Hispanic/Latino 


33. How many years have you been teaching at this school ? 
This is my first year 
2 to 3 years 
4 to 5 years 
6 to 1 0 years 
I I + years 

Native Hawaiian/Pacific Islander 
White/Caucasian 

Other (please specify): 


Thank you so much for taking time to complete this survey! 


Appendix D- 1 I 



Appendix E: 

Core Course Classification 


Appendix E 


Core Course Classification 


The core courses are: (1) English I-VI, (2) Algebra I and II, (3) Geometry, (4) Senior Math, 

(5) Life Science, (6) Physical Science, (7) Earth/Space Science, (8) Social Studies, (9) U.S. History, 
and (10) World Civilization. Within each core course domain, three types of classes are identified: 
(1) Comprehensive (COMP; regular), (2) Honors (HNR), and (3) others (e.g.. Advanced Placement 
[AP], Advanced [ADV], and Dual Credit). 


Table E-l. Breakdown by Core Courses and Class Type 


Credit Type 

Core Course 

Class Type 

Course Name 

(* added in Y3; ** added in Y5 ) 

Coding** 

English 

English (001) 

COMP (01) 

English 1 

00101 1 




Literacy Lab 1 

00101 1 




English 2 

001012 




ESL T rans2* 

001012 




Literacy Lab 2 

001012 




English 3 

001013 




English 4 

001014 




English Composition Lab 1 

001015 




English Composition Lab 2 

001016 



Honor (02) 

English 1 HNR 

001021 




English 2 HNR 

001022 




English 3 HNR 

001023 




English 4 HNR 

00 1 024 



Others (03) 

English Language and Composition AP 

001031 




English Literature and Composition AP 

001032 




English 1 ADV 

001033 




English 2 ADV 

001034 




English 3 ADV 

001035 




English 4 ADV 

001036 




English 4 Dual Credit 

001037 

Math 

Geometry (002) 

COMP (01) 

Geometry 

00201 1 




Geometry Lab 

0020 1 1 



Honor (02) 

Geometry HNR 

002021 



Others (03) 

Geometry ADV 

00203 1 




Accelerated Geometry** 

00203 1 


Algebra (003) 

COMP (01) 

Algebra 1 

00301 1 




Algebra Lab (Algebra 1 Lab) 

00301 1 




Algebra 2 

003012 




Algebra 2 Lab 

003012 




Algebra 1.5** 

003013 



Honor (02) 

Algebra 1 HNR 

003021 




Algebra 2 HNR 

003022 



Others (03) 

Algebra 2 ADV 

003031 
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Credit Type Core Course 

Class Type 

Course Name 

(* added in Y3; ** added in Y5 ) 

Coding** 

SR Math (004) 

COMP (01) 

Pre-Calculus 

0040 1 1 



Pre-Calculus Lab 

0040 1 2 



College Algebra 

0040 1 3 



Applied Mathematics 

004014 



Probability & Statistics** 

0040 1 5 



Technical Math** 

0040 1 6 


Honor (02) 

Pre-Calculus HNR 

00402 1 



College Algebra HNR 

004022 


Others (03) 

Pre-Calculus ADV 

00403 1 



Accelerated Pre-Calculus** 

00403 1 



Pre-Calculus MST 

004032 



Calculus AB AP 

004033 



Statistics AP 

004034 



College Algebra Dual Credit** 

004035 

Life Science (005) 

COMP (01) 

Anatomy/Physiology 

00501 1 



Biology 1 

005012 



Biology Special Topics* 

005012 



Biology 2 

005013 



Chemistry 1 

005014 


Honor (02) 

Anatomy/Physiology HNR 

005021 



Biology 1 HNR 

005022 



Chemistry 1 HNR 

005023 


Others (03) 

Biology 1 ADV 

00503 1 



Biology2 AP 

005032 



Biology AP* 

005032 



Chemistry 1 ADV 

005033 



Chemistry 2 ADV 

005034 



Chemistry AP* 

005035 

Earth/Space Science 

COMP (01) 

Integrated Science 1 

00601 1 

(006) 


Integrated Science IA 

00601 1 


Honor (02) 

Integrated Science IAHNR 

006021 


Other (03) 

Integrated Science IAADV 

00603 1 

Physical Science (007) 

COMP (01) 

Integrated Science 1 B 

0070 1 1 



Physics 1 

0070 1 2 


Honor (02) 

Integrated Science IB HNR 

007021 



Physics 1 HNR 

007022 


Others (03) 

Physics 1 ADV 

00703 1 



Physics B AP 

007032 



AP Physics LAIgebra-based** 

007032 



Integrated Science 1 B ADV* 

007033 

Social Studies (008) 

COMP (01) 

Exploring Civics 

00801 1 



Social Science 

008012 


Honor (02) 

Exploring Civics HNR 

00802 1 



Social Science HNR 

008022 


Others (03) 

Exploring Civics ADV 

00803 1 



Human Geography AP 

008032 



Social Science ADV 

008033 
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Credit Type Core Course 

Class Type 

Course Name 

(* added in Y3; ** added in Y5 ) 

Coding** 

I U.S. History (009) 

COMP (01) 

U.S. History 

00901 1 


Honor (02) 

U.S. History HNR 

00902 1 


Other (03 

U.S. History ADV 

00903 1 



U.S. History AP 

009032 

World Civilization (010) 

COMP (01) 

World Civilization 

01 001 1 


Honor (02) 

World Civilization HNR 

010022 


Other (03) 

World Civilization ADV 

010033 



World History AP* 

010034 
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Appendix F 


Core Course Analyses 


Table F-l. Pass Rates by Subject and Core Courses by School 


Subject 

Core Course 

Academy @ 
Shawnee 

n % 

Fern 

n 

Creek 

% 

Moore 
n % 

Valley 
n % 

Waggener 
n % 

Western 
n % 

Chi-square test 
X 2 df ES 

English 

Overall English 

686 

82.7 

2,071 

89.1 

1,064 

88.9 

1,080 

82.9 

1,195 

84.2 

840 

83.6 

46.26 

5 

0.03 


Geometry 

153 

73.9 

485 

80.2 

338 

87.3 

297 

75.4 

268 

63.4 

216 

51.9 

1 12.05 

5 

0.15 

Math 

Algebra 

282 

80.1 

758 

78.4 

525 

78.5 

444 

86.5 

418 

80.6 

401 

62.8 

74.80 

5 

0.05 


Senior Math 

22 

100.0 

358 

95.5 

205 

95.6 

124 

96.0 

139 

87.8 

1 18 

74.6 

66.53 

5 

0.13 


Overall Math 

457 

79.0 

1,601 

82.8 

1,068 

84.6 

865 

84.0 

825 

76.2 

735 

61.5 

187.51 

5 

0.10 


Life Science 

217 

49.3 

1,109 

75.9 

312 

88.1 

292 

82.2 

300 

85.3 

417 

65.2 

152.96 

5 

0.06 


Earth/Space Science 

164 

86.0 

373 

83.9 

292 

86.0 

301 

72.4 

182 

83.0 

299 

62.9 

73.08 

5 

0.14 


Physical Science 

157 

89.2 

36 

80.6 

272 

89.3 

298 

78.9 

196 

71.9 

185 

75.1 

34.63 

5 

0.12 


Overall Science 

538 

72.1 

1,518 

78.0 

876 

87.8 

693 

77.8 

678 

80.8 

901 

66.5 

128.91 

5 

0.04 


Social Studies 

272 

67.6 

1 12 

86.6 

327 

87.8 

229 

81.2 

210 

85.2 

281 

69.4 

61.03 

5 

0.003 

Social 

U.S. History 

89 

70.8 

355 

92.7 

167 

94.0 

262 

93.1 

204 

83.3 

194 

80.9 

57.91 

5 

0.03 

Studies 

World Civilization 

31 

90.3 

445 

81.3 

289 

94.5 

281 

94.3 

230 

82.2 

196 

80.1 

51.57 

5 

0.003 


Overall Social Studies 

392 

70.2 

912 

86.4 

783 

91.6 

111 

90.0 

644 

83.7 

509 

75.9 

149.82 

5 

0.01 

OVERAL 

.L 

2,073 


6,102 

84.3 

3,791 


3,608 

83.4 

3,342 


3,147 

71.9 


*Pearson chi-square tests. Effect sizes were estimated based on Somers’ d (Ferguson, 2009): A value of 0.2 is small effect size; a value of 0.5 is moderate, and a value of 0.8 is 
large. 


Appendix F- 1 


Table F-2. Trimester Schedule Status, Gender, LEP Status, and Grade Level Differences in Student Pass Rates 


Subject 

Core Course 

Trimester 
Schedule 
Status a 

Gender a 

LEP a 

9th Grade 

Grade Level a 
1 0th Grade 

1 Ith Grade 



P 

OR 

P 

OR 

P 

OR 

P 

OR 

P 

OR 

P 

OR 

English 

Overall English 

-1.19“ 

0.30 

0 . 68 *** 

1.97 

0.15 

1.17 

-1.15*** 

0.32 

-0.90*** 

0.41 

- 0 . 88 *** 

0.41 


Geometry 

- 0.86 

0.42 

0.16 

1.17 

-0.08 

0.92 

0.33 

1.39 

0.10 

1.10 

-1.18 

0.31 

M?a+h 

Algebra 

-20.40 

0.00 

0.50“ 

1.65 

-0.12 

0.88 

0.10 

1.10 

0.78* 

2.19 

- 0.02 

0.98 


Senior Math 

-18.77 

0.00 

0.60 

1.82 

-0.78 

0.46 

-26.84 

0.00 

- 

- 

- 1 . 68 *” 

0.19 


Overall Math 

-2.57*** 

0.08 

0.41*** 

1.51 

-0.004 

1.00 

-0.98*** 

0.38 

-1.04*** 

1.35 

-1.09*** 

0.34 


Life Science 

-0.92* 

0.40 

0.50*** 

1.64 

0.18 

1.20 

-0.09 

0.91 

-I 49 *** 

0.23 

-0.17 

0.84 


Earth/Space Science 

2.03**" 

7.63 

0.58” 

1.79 

0.39 

1.47 

- 20.21 

0.00 

-22.04 

0.00 

1.27 

3.56 

UCI ILC 

Physical Science 

-0.12 

0.89 

0.60* 

1.82 

-19.78 

0.00 

-1.80* 

0.17 

-0.72 

0.49 

-0.53 

0.59 


Overall Science 

0.04 

1.04 

0.52*** 

1.67 

0.15 

1.17 

-0.46* 

0.63 

-1.09*** 

0.38 

-0.29 

0.75 


Social Studies 

_ | 2 ^ 

0.28 

0.69”* 

1.99 

0.04 

1.04 

-21.00 

0.00 

-21.32 

0.00 

-21.82 

0.00 

Social 

U.S. History 

- 1 .62*** 

0.19 

0.54” 

1.71 

1.09 

2.98 

- 

- 


0.06 

-0.72”* 

0.00 

Studies 

World Civilization 

-2.47* 

0.09 

0.51* 

1.67 

0.29 

1.34 

-21.93 

0.00 

-20.04 

0.00 

-20.34 

0.00 


Overall Social Studies 

-1.59*** 

0.20 

0.63*** 

1.88 

0.24 

1.28 

-0.97*** 

0.38 

-0.45* 

0.64 

-0.76** 

0.47 

OVERALL 

-1 .26*** 


0.53 

1.69 

0.1 1 

1.12 

-0.95*** 

0.39 

-0.99*** 

0.37 

-0.85*** 



Note, p = coefficient; OR = the odds of passing the course was [odds ratio] times higher (or lower) if the student was a “non -reference group” in comparison with “the reference 
group”. 

a Three-term courses, male, students with LEP, and 12th graders are the reference groups. 

— the ratio of the two groups was too large for comparison with numeric errors. 

***p < 0.00 1 
**p < 0.0 1 

*p < 0.05 
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Table F-3. FRPM and Race Differences in Student Pass 


Subject 

Core Course 

FRPM Status 

i- am . n Reduced-Price 

Free Meals . . 

Meals 

Asian ° 

Race/Ethnicity 
Black ° Hispanic a 

Other ° 



P 

OR 

P 

OR 

P 

OR 

P 

OR 

P 

OR 

P 

OR 

English 

Overall English 

-0.33* 

0.72 

0.24 

1.27 

0.95 

2.58 

-0.20 

0.82 

-0.05 

0.96 

0.28 

1.32 


Geometry 

-0.51* 

0.60 

-0.07 

0.94 

1.36 

3.88 

-0.40* 

0.67 

0.98* 

2.66 

0.18 

1.20 

Ma+h 

Algebra 

-0.34 

0.72 

0.17 

1.19 

2.22 

9.16 

-0.31* 

0.74 

0.31 

1.36 

0.36 

1.43 


Senior Math b 

-0.39 

0.68 

0.21 

1.24 

- 

- 

-0.83* 

0.44 

-1.14** 

0.32 

-0.65 

0.52 


Overall Math 

-0.58*“ 

0.60 

0.07 

1.07 

1.67** 

5.33 

-0.48“* 

0.62 

0.42* 

1.52 

0.24 

1.27 


Life Science 

-0.64*“ 

0.53 

-0.13 

0.88 

1.09* 

2.97 

-0.32* 

0.73 

0.52* 

1.68 

0.90 

2.47 


Earth/Space Science 

-0.44 

0.64 

0.35 

1.42 

0.18 

1.20 

-0.51** 

0.59 

0.56 

1.74 

-0.13 

0.88 

1 Ul IL(J 

Physical Science 

0.31 

1.36 

-0.21 

0.81 

0.78 

2.18 

-0.37 

0.69 

1.93 

6.86 

1.09 

2.97 


Overall Science 

-0.36“ 

0.70 

-0.01 

0.99 

0.71 

2.04 

-0.35“ 

0.70 

0.61” 

1.84 

0.51 

1.66 


Social Studies 

-1.20*** 

0.30 

-0.10 

0.90 

1.40 

4.06 

0.23 

1.25 

1.22“ 

3.40 

0.75 

2.12 

Social 

U.S. History 

-0.54* 

0.58 

-0.24 

0.78 

- 

- 

-0.01 

1.00 

0.16 

1.18 

- 

- 

Studies 

World Civilization 

-0.25 

0.78 

-0.11 

0.89 

20.51 

0.00 

-0.13 

0.88 

0.03 

1.03 

-0.01 

0.99 


Overall Social Studies 

-0.69*“ 

0.50 

-0.13 

0.88 

2.59** 

13.38 

0.03 

1.03 

0.58* 

1.79 

0.75 

2.1 1 

OVERALL 

- 

-0.44*** 


0.06 

1.0 1 

I2|*** 

3.34 

-0.27*** 

0.76 

0.38“ 

1.46 

0.42 



Note. Some odds ratios are extremely large because of the extreme unbalance sample size between groups. fS = coefficient; OR = the odds of passing the course was [odds ratio] 
times higher (or lower) if the student was a “non-reference group” in comparison with “the reference group”. 
a Students with paid meal status and White students are the reference groups. 

— the ratio of the two groups was too large for comparison with numeric errors. 

***p < 0.00 1 
**p < 0.0 1 
*p < 0.05 


Appendix F-3 


Appendix G: 

Means and Standard Deviations of Student Perceptions of the 
Academic Challenge Item Index by School 


Appendix G 


Means and Standard Deviations of Student Perceptions of the Academic Challenge Item Index by School 



All Schools 

Academy @ 
Shawnee 

Fern Creek 


4oor€ 

i 


Valley 


Waggener 

Western 


n 

M 

SD 

n 

M 

SD 

n 

M 

SD 

n 

M 

SD 

n 

M 

SD 

n 

M 

SD 

n 

M 

SD 

B2 

4,637 

2.56 

0.85 

350 

2.43 

0.93 

1,378 

2.52 

0.79 

803 

2.46 

0.78 

871 

2.73 

1.07 

717 

2.60 

0.74 

518 

2.52 

0.71 

E2 

4,580 

2.91 

0.64 

338 

2.86 

0.76 

1,365 

3.00 

0.56 

798 

2.82 

0.70 

859 

2.90 

0.64 

71 1 

2.92 

0.66 

509 

2.90 

0.63 

E3 

4,571 

2.68 

0.76 

337 

2.44 

0.91 

1,365 

2.78 

0.70 

794 

2.62 

0.76 

859 

2.73 

0.74 

709 

2.60 

0.80 

507 

2.66 

0.74 

CSS Mean 

4,643 

2.71 

m 

351 

2.56 

m 

1,380 

2.76 

m 

805 

2.63 

EE3 

871 

2.78 

m 

718 

2.70 

m 

518 

2.68 

EES 


Note. B2 = “I think school is fun and challenging”; E2 = “My teachers provide academically challenging content”; E3 = “Teachers at my school assign meaningful homework on 
a regular basis.” 
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Teacher Survey Aggregate Means, Standard Deviations, and 
Percentages for Scales by Item 
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Teacher Survey Aggregate Means, Standard Deviations, and Percentages for Scales by Item 


Item 

n 

M 

SD 



Agree 

Strongly 

Agree 

Student Academic Engagement 

Teachers at my school challenge students academically. 

86 

2.93 

0.59 

1.2% 

1 7.4% 

68.6% 

1 2.8% 

Teachers at my school encourage students to go beyond stated 
expectations. 

86 

2.97 

0.66 

1.2% 

1 9.8% 

60.5% 

1 8.6% 

Teachers at my school involve students in setting expectations. 

86 

2.73 

0.64 

1.2% 

33.7% 

55.8% 

9.3% 

Teachers at my school involve students in planning lessons. 

86 

2.17 

0.65 

1 0.5% 

65.1% 

20.9% 

3.5% 

Teachers at my school involve students in developing criteria for 
assessing their assignments. 

86 

2.44 

0.66 

4.7% 

5 1 .2% 

39.5% 

4.7% 

Teachers at my school work with disenfranchised students to help 
them feel more connected to school. 

86 

3.07 

0.59 

-- 

14.0% 

65.1% 

20.9% 

Scale Total 


16.31 

2.72 






Student Academic Challenge 


In general, students seem to enjoy my school. 

86 

2.72 

0.64 

3.5% 

27.9% 

6 1 .6% 

7.0% 

In general, high grades at my school are viewed as an indicator of easy 
course work. 

86 

2.34 

0.59 

2.3% 

65.1% 

29.1% 

3.5% 

In general, students seem to be bored at my school. 

86 

2.38 

0.58 

1.2% 

62.8% 

32.6% 

3.5% 

In general, students at my school perceive challenge as a way to go 
beyond expectations. 

86 

2.24 

0.72 

1 2.8% 

53.5% 

30.2% 

3.5% 

In general, high grades at my school are viewed as an indicator of 
mastering high standards. 

86 

2.84 

0.67 

3.5% 

20.9% 

64.0% 

1 1.6% 

In general, students do well in my school. 

86 

2.55 

0.61 

3.5% 

40.7% 

53.5% 

2.3% 

Scale Total 


15.07 

2.09 





Teacher Collaboration and Support 

Teachers have sufficient time to collaborate with colleagues to plan 
and improve instruction. 

94 

2.83 

0.79 

3.2% 

18.1% 

58.5% 

20.2% 

Collaboration is seen as an important part of this school's culture. 

94 

3.28 

0.65 

2.1% 

4.3% 

57.4% 

36.2% 

Teachers have sufficient resources to provide quality instruction. 

94 

2.96 

0.72 

8.5% 

14.9% 

6 1 .7% 

14.9% 
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Item 

n 

M 

SD 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Teachers are generally willing to try new ideas. 

94 

3.13 

0.47 

- 

5.3% 

76.6% 

18.1% 

Teachers in this school trust each other. 

94 

2.84 

0.61 

2.1% 

21.3% 

67.0% 

9.6% 

It is okay in this school to discuss with other teachers positive and/or 
negative feelings, concerns, and frustrations related to instruction and 
practice. 

94 

2.98 

0.60 

1.1% 

1 6.0% 

67.0% 

1 6.0% 

Teachers discuss difficulties and work together to develop solutions. 

94 

3.02 

0.62 

- 

18.1% 

6 1 .7% 

20.2% 

Teachers support each other's efforts to make improvements. 

94 

3.10 

0.55 

- 

1 0.6% 

69.1% 

20.2% 

Access to expertise and resources is available and provided in a 
timely manner. 

94 

2.87 

0.66 

3.2% 

19.1% 

64.9% 

1 2.8% 

Scale Total 


27.00 

3.69 





Teacher Self-Efficacy j 

How often . . . 

n 

M 

SD 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

does the emphasis on success discourage you from trying new 
approaches? 

87 

2.59 

1.03 

21.8% 

16.1% 

43.7% 

1 8.4% 

do you feel that you are able to work effectively? 

87 

3.48 

0.59 

- 

4.6% 

42.5% 

52.9% 

are you satisfied with the quality of your work? 

87 

3.55 

0.57 

- 

3.4% 

37.9% 

58.6% 

do you feel that you are being successful in your work? 

87 

3.44 

0.59 

- 

4.7% 

46.5% 

48.8% 

do you have sufficient self-confidence to defend your own points of 
view? 

87 

3.69 

0.52 

- 

2.3% 

26.7% 

70.9% 

do you feel adequately prepared to use formative data (e.g., class 
work, homework, and other instructional activities) to adjust your 
instruction? 

87 

3.52 

0.57 

- 

3.4% 

41.4% 

55.2% 

do you feel adequately prepared to use summative data (e.g., end of 
chapter, unit, or course assessments) to adjust your instruction? 

87 

3.51 

0.59 

- 

4.6% 

40.2% 

55.2% 

do you feel confident in teaching all required content standards in 
your subject area? 

87 

3.61 

0.65 

1.1% 

5.7% 

24.1% 

69.0% 
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Item 

n 

M 

SD 



Agree 

Agree 

do you feel confident in adjusting your instructional practices to meet 
the needs of individual students? 

87 

3.48 

0.61 

-- 

5.7% 

40.2% 

54.0% 

do you worry about being criticized if positive results are not readily 
apparent? 

87 

2.82 

1.01 

1 2.6% 

24.1% 

32.2% 

31.0% 

Scale Total 


33.73 

3.62 





Perceptions of Professional Learning Communities 

1 feel that my contribution to the PLC is valued. 

Successful results with students are a regular part of discussions 
during PLCs. 

Teacher collaboration related to instruction has increased as a result 

92 

92 

92 

3.16 

3.08 

3.03 

0.70 

0.73 

0.78 

2.2% 

2.2% 

5.4% 

1 0.9% 
1 6.3% 
1 2.0% 

55.4% 

53.3% 

56.5% 

31.5% 

28.3% 

26.1% 

of my school’s PLCs. 

The PLC facilitator is adequately prepared for all PLC meetings. 

92 

3.14 

0.76 

5.4% 

6.5% 

56.5% 

31.5% 

School administrators (e.g., principal and assistant principal) attend 
the PLC meetings. 

91 

2.60 

0.99 

1 9.8% 

1 6.5% 

47.3% 

1 6.5% 

1 feel that my instructional practice has improved as a result of 
participating in PLCs. 

92 

2.92 

0.83 

6.5% 

1 8.5% 

51.1% 

23.9% 

Time for collaborative planning is uninterrupted. 

92 

2.50 

0.87 

1 5.2% 

29.3% 

45.7% 

9.8% 

New ideas are welcomed and supported. 

92 

3.10 

0.61 

-- 

14.1% 

62.0% 

23.9% 

The PLC facilitator encourages all voices to be heard. 

92 

3.24 

0.70 

3.3% 

5.4% 

55.4% 

35.9% 

Scale Total 


23.35 

4.83 






PLC Big ldea#l: Building a Collaborative Culture 


Collaboration focuses on the group learning together. 

88 

3.07 

0.68 

3.4% 

9.1% 

64.8% 

22.7% 

The PLC team is focused on critical questions. 

88 

3.06 

0.67 

1.1% 

1 5.9% 

59.1% 

23.9% 

Products of collaboration efforts are explicit (e.g., norms, goals, 
outcomes, assessments, and lesson plans). 

88 

3.18 

0.64 

-- 

1 2.5% 

56.8% 

30.7% 

Subscale Total 


9.31 

2.88 






PLC Big Idea #2: Ensuring All Students Learn 


The PLC identifies/discusses essential common outcomes. 

88 

3.23 

0.56 

1.1% 

3.4% 

67.0% 

28.4% 

Outcomes align with content standards/pacing guides. 

88 

3.31 

0.61 

-- 

8.0% 

53.4% 

38.6% 
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Item 

n 

M 

SD 

Strongly 

Disagree 

Disagree 

Agree 

Agree 

Instructional plans include specific learning targets. 

88 

3.33 

0.62 

-- 

8.0% 

51.1% 

40.9% 

PLC teams develop multiple common formative assessments/tasks. 

88 

3.23 

0.71 

2.3% 

9.1% 

52.3% 

36.4% 

Instructional plans and activities include strategies to increase the use 
of higher order thinking skills. 

88 

3.10 

0.71 

1.1% 

1 7.0% 

52.3% 

29.5% 

Instructional plans provide support for students at different learning 
levels. 

88 

3.03 

0.70 

1.1% 

1 9.3% 

54.5% 

25.0% 

Subscale Total 


19.23 

3.25 






PLC Big Idea #3: Establishing a Focus on Results 


PLCs pursue specific and measurable team performance goals 
(S.M.A.R.T.). 

PLCs access relevant student information to monitor progress 
toward goals. 

88 

88 

3.19 

3.26 

0.62 

0.62 

1.1% 

1 1 .4% 
5.7% 

58.0% 

59.1% 

30.7% 

34.1% 

PLCs utilize common assessments and formative tasks to monitor 
student progress. 

88 

3.31 

0.68 

2.3% 

5.7% 

51.1% 

40.9% 

PLCs reflect on data to identify students in need of intervention 
(enrichment) and plans for strategies, activities, and interventions. 
PLCs utilize an increasingly directive, timely, and systematic response 
to students not meeting established outcomes and learning targets. 

88 

88 

3.27 
3.1 1 

0.64 

0.73 

1.1% 

2.3% 

6.8% 

14.8% 

55.7% 

52.3% 

36.4% 

30.7% 

Subscale Total 


16.15 

2.88 





PLC Big Ideas Scale Total 


44.68 

7.32 






Course Assignment Process 


My school has a course assignment process that results in the best 
placement for students. 

87 

2.43 

0.87 

1 7.2% 

3 1 .0% 

43.7% 

8.0% 

My school has a clear process for identifying students early when they 
are struggling. 

87 

2.62 

0.81 

9.2% 

3 1 .0% 

48.3% 

1 1.5% 

My school has effective practices in place to successfully intervene 
with students who are struggling in a course. 

87 

2.68 

0.83 

9.2% 

27.6% 

49.4% 

1 3.8% 

My school has effective practices in place to move students up to a 
higher-level course when they have been successful. 

87 

2.67 

0.80 

8.0% 

29.9% 

49.4% 

1 2.6% 

Scale Total 


10.39 

2.79 
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Item 

n 

M 

SD 

Never 

Seldom 


Always 

How Often Do CAT Lessons Follow the Elements of the Classroom Instructional Framework? 


Establishing engagement 
Fostering connections 
Deepening understanding 
Making meaning 


42 

42 

42 

42 


3.12 

0.94 

9.5% 

3.21 

0.98 

1 1.9% 

2.88 

1.02 

14.3% 

2.98 

0.95 

1 1.9% 


9.5% 

40.5% 

40.5% 

2.4% 

38.1% 

47.6% 

14.3% 

40.5% 

3 1 .0% 

9.5% 

47.6% 

3 1 .0% 


Scale Total 




CAT Preparation and Impact 


1 have received sufficient training and coaching to successfully implement 
CAT. 

42 

2.86 

0.90 

9.5% 

1 9.0% 

47.6% 

23.8% 

The CAT lessons provided to me can be sufficiently covered during the 
allotted time. 

42 

2.86 

0.87 

1 1.9% 

9.5% 

59.5% 

1 9.0% 

1 feel equipped to address social and emotional issues that arise during CAT. 

42 

2.98 

0.84 

9.5% 

7.1% 

59.5% 

23.8% 

1 believe that the CAT advisory periods are beneficial to students. 

42 

2.64 

1.03 

1 9.0% 

1 9.0% 

40.5% 

21.4% 

1 receive the CAT lessons in time to adequately prepare for the lesson. 

42 

2.83 

0.82 

9.5% 

14.3% 

59.5% 

1 6.7% 

CAT advisory periods increase students’ college readiness. 

42 

2.36 

0.93 

1 9.0% 

38.1% 

3 1 .0% 

1 1 .9% 

Overall, CAT advisory periods are making a positive contribution to the 
success of our students. 

42 

2.62 

0.94 

1 1.9% 

33.3% 

35.7% 

1 9.0% 

Scale Total 


19.14 

5.34 
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